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Overview 



In 2004, Lumina Foundation for Education launched “Achieving the Dream: Community Colleges 
Count,” a national initiative aimed at improving success among community college students, particularly 
low-income students and students of color. Now encompassing more than 130 institutions in 24 states and 
the District of Columbia, Achieving the Dream helps community colleges huild a “culture of evidence” hy 
using student records and other data to examine students’ performance over time and to identify harriers to 
academic progress. From there, community colleges are expected to develop intervention strategies 
designed to improve student outcomes; conduct further research on student progress; and bring effective 
programs to scale. As a result, it is anticipated that colleges will see measurable improvements over time 
in student outcomes, including increased progress through developmental education and college-level 
“gatekeeper” (introductory) courses, grades, persistence, and completion of credentials. 

This report examines the first 26 colleges to join Achieving the Dream in 2004 (known as the “Round 
1” colleges), and tracks their progress through spring 2009. The key findings are: 

• Four out of five Round 1 colleges adopted practices associated with a moderate to strong 
culture of evidence. These colleges made important enhancements to their institutions, includ- 
ing more sophisticated methods for data analysis and more efficient systems for monitoring 
their efforts to improve students’ achievement. Conversely, about one-fifth of the colleges still 
struggled to implement many of the initiative’s recommended practices, hindered primarily by 
weak institutional research capacity. 

• Colleges that made the greatest strides shared several key characteristics, including broad- 
based involvement of college administrators, faculty, and staff; strong institutional research de- 
partments that produced accessible reports on student achievement; regular evaluations of their 
programs; and scale-up of successful programs. 

• Colleges instituted a wide range of strategies to improve student achievement, hut a ma- 
jority of them remained small in scale. The most popular strategies were tutoring, supple- 
mental instruction, advising, success courses, and learning communities. However, a majority 
of these reforms reached less than 10 percent of their intended target populations. 

• Achieving the Dream had an important influence on most colleges. Representatives from 
three-fourths of the colleges said that the initiative had at least some influence on their devel- 
opment of a culture of evidence. Other important influences included accreditation systems, 
grants in addition to those from Achieving the Dream, and visionary college leaders. 

• Trends in student outcomes remained relatively unchanged, with a few exceptions. On aver- 
age, after Achieving the Dream was introduced, colleges saw modest improvements in the per- 
centage of students completing gatekeeper college English courses and courses completed. In 
contrast, students’ persistence and the percentage of students completing developmental math, 
developmental English, developmental reading, and gatekeeper math courses remained substan- 
tially the same. 



A final report on the Round 1 colleges, with a follow-up on the trends in student outcomes, is planned 
for 2012-2013. 
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Preface 



Today, more students than ever are enrolling in community college, but nearly half of them never go on 
to earn a degree or certificate. Until recently, most community colleges did not have the capacity or the 
incentives to gather information about their students’ performance over time and the possible barriers to 
their success. Traditionally focused on increasing access to postsecondary education, particularly for 
low-income students, community colleges are now turning more attention to improving the academic 
success of their students. But without concrete evidence to illuminate how and why their students are 
underperforming, colleges have been hard-pressed to take steps to help them improve. 

Enter “Achieving the Dream: Community Colleges Count,” an ambitious national initiative launched by 
Lumina Foundation for Education in 2004, with support from a number of other philanthropies and 
organizations. The initiative helps community colleges learn how to collect and analyze student 
performance data in order to build a “culture of evidence” — a culture in which colleges routinely use 
solid evidence to develop institutionwide reform strategies that are aimed at helping their students 
succeed academically. Achieving the Dream has grown into a national movement; more than 130 
community colleges are now participating. 

This interim report takes stock of the experiences among the first 26 colleges from five states to join the 
initiative (the “Round 1” colleges). It offers a look at the influence of the initiative in its early stages, while 
its practices and supports were still evolving. The study shows that after five years in Achieving the 
Dream, four out of five of the 26 Round 1 colleges had adopted some or all of the practices associated 
with a culture of evidence. While the colleges instituted a wide range of strategies to help their students 
succeed, increasing student achievement and persistence levels in the colleges remains a huge chal- 
lenge. 

What are we to make of Achieving the Dream at this juncture? On the one hand, it represents an unprece- 
dented effort to help community colleges change their institutional cultures so they can continuously 
pinpoint where, how, and why their students are underperforming, and then implement reforms to help 
them improve. Achieving the Dream was at the vanguard, anticipating — and helping spur — the growing 
national focus on institutional accountability in community colleges. After five years in the initiative — 
the first two years of which were spent planning and setting up strategies for reform — a majority of the 
Round 1 colleges are collecting data on their students’ performance and are committed to using those data 
to design and implement effective reforms. On the other hand, trends in student outcomes — such as 
course completion, persistence, maintaining good grades, and earning college credentials — have re- 
mained relatively unchanged at these colleges, underscoring the depth of the challenge facing all commu- 
nity colleges. 

One lesson from Achieving the Dream is that simply understanding the problems is not enough to solve 
them. Nevertheless, Achieving the Dream has provided a better sense of where the colleges have to go to 
improve and how to get there, creating an infrastmcture through which change can occur. What remains is 
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for colleges to harness the potential power of that infrastructure — and for Achieving the Dream itself to 
further refine its approach in order to better help colleges focus systematically on the issues that affect 
student achievement. 

Achieving the Dream is not a static intervention, hut a living, organic process — and many of the Round 1 
colleges are now moving into the next stage of that process. A final report on the Round 1 colleges is 
planned for 2012-2013, hy which time they will have had more time to institutionalize their reforms. 



Gordon Berlin 
President 
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Executive Summary 



Across the United States, enrollments at community colleges are soaring. In fall 2009, 
approximately 8 million students took courses for credit — nearly a 17 percent increase from 
two years before.' Community colleges tend to be welcoming and accessible institutions, par- 
ticularly for nontraditional, low-income, and minority students. But while community colleges 
do a good job of helping students get a foothold in higher education, close to half of the students 
who begin at community colleges with the intention of earning a certificate or degree do not 
achieve their goal and are not enrolled in any college or university after six years.^ A growing 
number of policymakers and educators are calling upon community colleges to do better. 

“Achieving the Dream: Community Colleges Count” is a national initiative that was 
launched in 2004 and was designed to increase the academic success of community college stu- 
dents, with a special emphasis on low-income students and students of color. Its approach is to 
help community colleges build a “culture of evidence” by using student records and other data 
to examine how students are performing over time and to identify barriers to academic progress. 
From there, community colleges are expected to develop intervention strategies designed to im- 
prove student outcomes, conduct further research on student progress, and bring effective pro- 
grams to scale. Lumina Foundation for Education launched Achieving the Dream and was its 
principal funder, with a large group of philanthropies and other organizations joining Lumina to 
support the initiative. Today, more than 130 community colleges in 24 states and the District of 
Columbia are involved in Achieving the Dream (see Figure ES.l). 

MDRC — a nonprofit, nonpartisan education and social policy research organization 
— and the Community College Research Center (CCRC) are evaluating the work of the first 26 
colleges to join the initiative from Elorida, New Mexico, North Carolina, Texas, and Virginia 
(called the “Round 1” colleges here). This report covers a five-year period, beginning in the 
2004-2005 academic year. Because the process of building a culture of evidence is envisioned 
as a long-term effort, a future report will examine trends in student achievement after the col- 
leges have had more time to institutionalize reforms that began under Achieving the Dream. 

Overall, the report concludes that: 



'American Association of Community Colleges (AACC), Community College Enrollment Survey: An 
Analysis of Estimated Fall 2009 Headcount Enrollments at Community Colleges (Washington, DC: AACC, 
2009). 

^Stephen Provasnik and Michael Planty, Community Colleges: Special Supplement to the Condition of 
Education 2008, NCES 2008-033 (Washington, DC; National Center for Education Statistics, Institute of Edu- 
cation Sciences, U.S. Department of Education, 2008). 
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Achieving the Dream: Community Colleges Count 



Figure ES.l 

States and Colleges Involved in Achieving the Dream, 2004 to 2010 




■ Round 1 Achieving the Dream states (joined in 
2004-2005) 

□ Achieving the Dream states that joined after 2005 

A Round 1 Achieving the Dream colleges (joined in 
2004-2005) 

• Colleges that joined Achieving the Dream after 2005 




NOTE: Multi-college districts are represented with one dot on this map. 
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• Many Round 1 colleges made important strides in building a stronger culture of evi- 
dence during their five-year participation in the initiative. In particular, colleges en- 
hanced their leadership commitment to student success, increased their research capaci- 
ty, and developed a number of interventions aimed at improving their students’ 
achievement. 

• By spring 2009, four out of five Round 1 colleges had adopted a number of practices 
associated with a strong culture of evidence, with one-fifth of the colleges still facing 
major challenges in undertaking this work. 

• While colleges instituted a wide range of strategies to improve student achievement un- 
der the auspices of Achieving the Dream, a majority of these reforms reached less than 
10 percent of their intended target populations — likely too few to make demonstrable 
progress on improving student achievement overall. 

• Other forces besides Achieving the Dream sometimes helped the colleges build a cul- 
ture of evidence, including accreditation processes and grants other than those from the 
initiative. However, about three-fourths of the colleges indicated that Achieving the 
Dream had at least some influence in helping them develop a culture of evidence. 

• Trends in student outcomes remained relatively unchanged, except for modest im- 
provements in gatekeeper (introductory) college English courses and the completion of 
courses attempted within the first two years. 



The Theory of Action Behind Achieving the Dream 

From the earliest conversations, it was clear that Lumina Foundation and the partners 
had big ambitions for Achieving the Dream. They spoke about fostering fundamental changes 
in the culture and operations of community colleges — changes that would lead to measureable 
and lasting improvements in student outcomes. They also spoke about changing the context in 
which community colleges operate — everything from the rules governing their accreditation 
and funding to public awareness and support for community colleges. 

On the ground, participating colleges were expected to enact a five-step process of insti- 
tutional reform, which included (1) securing leadership commitment; (2) using data to prioritize 
actions; (3) engaging stakeholders; (4) implementing, evaluating, and improving intervention 
strategies; and (5) establishing a culture of continuous improvement. To help colleges institute 
their reforms. Achieving the Dream provided them with both monetary and technical support. 
The Round 1 colleges were awarded an initial year-long planning grant of $50,000 each, fol- 
lowed by annual grants of $100,000 for four years ($450,000 total). In addition, the colleges 
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were aided by two consultants: a data facilitator, who helped them perform the data collection 
and analysis and interpret the results; and a coach, who helped them set priorities, build consen- 
sus, and implement strategies for improvement. The initiative also sponsored a Kick-off Insti- 
tute and annual Strategy Institutes for all the Achieving the Dream colleges, where they learned 
more about the Achieving the Dream process, made plans for their own campuses, and shared 
ideas and lessons with other colleges on how to help students be more successful. Colleges also 
received technical assistance from the initiative’s eight founding partner organizations, led by 
MDC, Inc., the managing partner of Achieving the Dream. 

All Achieving the Dream colleges were asked to try to “move the needle” on five mea- 
sures of student success: (1) completion of developmental courses and progression to credit- 
bearing courses; (2) completion of so-called gatekeeper courses, including introductory college 
courses in English and math; (3) completion of attempted courses with a grade of “C” or better; 
(4) persistence from semester to semester and from year to year; and (5) attainment of college 
credentials. Colleges were expected to track these outcomes for each fall cohort of entering stu- 
dents and to make comparisons with past cohorts to determine whether the outcomes improved 
over time. As noted above, colleges were also expected to break down the results for subgroups 
of students defined by race, ethnicity, and income. 

Throughout the history of Achieving the Dream, there was some tension between those 
who believed that gains in student outcomes should come quickly and those who believed it 
was a long-term process. The initiative’s “Integrated Action Plan” was vague on this point, in- 
dicating that colleges should see “measurable improvement in success rates” after four years 
and have “achieved their long-term targets for student success” after eight years. What was clear 
was that the five-step process was iterative, so any improvements in student outcomes depended 
on the colleges’ ability to master the initial steps. Even if colleges did all that was asked of 
them, it could still take time for improvements to show up in the data, given the longitudinal 
nature of the measures. Some of the partners cited the oft-used metaphor of turning a ship to 
describe the work of Achieving the Dream. On the one hand, they firmly believed that the initia- 
tive would set community colleges on a better course; on the other hand, they recognized that 
changes would likely occur only gradually and might not be fully apparent for several years’ 
time. 



In analyzing the work and progress of the Round 1 colleges, it is also important to note 
that these schools began their work in Achieving the Dream as the initiative was still evolving. 
Indeed, since its beginning in 2004, the initiative has expanded, modified, and codified many of 
its practices and supports, with the Round 1 colleges receiving less intensive versions of these 
messages during the early phases of their implementation. As such, the Round 1 colleges’ 
progress should be seen as offering a look at the initiative’s influence during its early develop- 
ment, while its practices and supports were still being solidified. 
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The Round 1 colleges are diverse in size, location, and student characteristics. The larg- 
est institution is Houston Community College, which had a full-time equivalent 12-month 
enrollment of over 32,000 students in 2008-2009. The smallest institution is Martin Community 
College in North Carolina, which had a full-time equivalent enrollment of 410 students. The 
colleges are located in large and midsize cities, suburbs, and small towns. White students make 
up a majority or plurality of students at most of these institutions, but nearly all the colleges 
enrolled substantial numbers of African American, Hispanic, or Native American students. 



How Was the Research Conducted at the Round 1 Colleges? 

A research team comprising MDRC and CCRC staff studied the Round 1 colleges over 
several years to learn how they implemented Achieving the Dream and whether they made 
progress in developing a culture of evidence on campus. The team visited each of the 26 colleg- 
es two times: once during the first year of implementation, in spring 2006, and again during the 
last year of implementation, either in fall 2008 or spring 2009. The visits lasted two to three 
days and included interviews with the administrators, faculty members, and institutional re- 
search staff, among others. The research team also conducted a survey of college administrators 
and faculty at 23 of the 26 colleges in 2007-2008. In addition, the research team reviewed an- 
nual reports that the colleges submitted to Achieving the Dream, as well as reports prepared by 
the coaches and data facilitators. Finally, the research team analyzed student records data that 
the colleges submitted to the Achieving the Dream database, which allows researchers to track 
student performance over time. The analysis examines student outcomes at the institutional lev- 
el for students who began college each fall between 2002 and 2007 (three years before the initi- 
ative began and three years after its implementation). Each entering cohort is tracked for two 
years. 



Have the Round 1 Colleges Created a Culture of Evidence? 

The bulk of this report assesses the Round 1 colleges’ success between spring 2006 and 
spring 2009 in developing a culture of evidence — that is, their progress in instituting the five 
main steps of the Achieving the Dream model, described above. 

As Table ES.l shows, most of the Round 1 colleges made substantial progress in devel- 
oping more evidence-based systems aimed at improving student success during their participa- 
tion in Achieving the Dream. Eighty-one percent (21) of the Round 1 colleges improved their 
culture of evidence over the course of their five-year participation in the initiative, and 19 per- 
cent (five) of the colleges continued to have major challenges implementing the initiative’s 
model for institutional improvement. Indeed, most of the Round 1 colleges made a number of 
enhancements to their institutions, including the use of more sophisticated methods for data 
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Achieving the Dream: Community Colleges Count 
Table ES.l 



Culture of Evidence at Achieving the Dream Round 1 Colleges, 
Spring 2006 and Spring 2009 





Spring 2006 


Spring 2009 


Culture of Evidence Rating 


Number of 
Round 1 Colleges 


Percentage of 
Round 1 Colleges 


Number of 
Round 1 Colleges 


Percentage of 
Round 1 Colleges 


Strong or very strong 
culture of evidence 


6 


23 


11 


42 


Some culture of evidence 


5 


19 


10 


38 


Weak or very weak 
culture of evidence 


15 


58 


5 


19 


Total 


26 


100 


26 


100 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives, 
Achieving the Dream coaches, and data facilitators. 

NOTE: Percentages may not add to 100 because of rounding. 

analysis, numerous interventions aimed at increasing students’ success, and more efficient sys- 
tems for monitoring their efforts to improve students’ achievement. The colleges can be divided 
into three categories, based on their success in institutionalizing a culture of evidence. 

Strong Culture of Evidence 

Eleven Round 1 colleges (42 percent) had implemented most of the practices that are 
associated with a strong culture of evidence by the end of their five-year participation in 
Achieving the Dream. These colleges had excelled at building or further strengthening all five 
aspects of the initiative’s institutional improvement model — that is, they (1) had strong leader- 
ship systems committed to improving student success; (2) had well-developed institutional re- 
search departments that conducted intensive, in-depth analyses of their students’ achievement; 
(3) involved faculty, staff, students, and external stakeholders in multiple aspects of their reform 
agenda; (4) had used well-developed strategies, with one or more strategies reaching substantial 
proportions of their target population; and (5) had instituted multiple processes for strategic 
planning and program review, and made financial and programmatic decisions based on evi- 
dence of improved student achievement. As described in Box ES.l, the colleges that made the 
greatest strides in developing an evidence-based approach to improving student achievement 
shared several important characteristics, including the broad-based involvement of college ad- 
ministrators, faculty, and staff; strong institutional research departments that produced reports 
on student achievement that people could readily understand; regular evaluations of interven- 
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Box ES.1 



“Star” Colleges in Achieving the Dream: Institutions that Made 
Solid Progress in Building a Culture of Evidence 

Four of the 26 Round 1 colleges stood out because they had a weak culture of evidence at 

the start of the initiative and made great strides in building up their culture over five years. 

These colleges made changes that set them apart from the other colleges that made less 

progress, including: 

• Multiple senior administrators who were deeply involved in the colleges’ reforms. 

Each college had a cadre of senior leaders, including the vice presidents of instruction 
and student services and the head of institutional research (IR), leading its Achieving 
the Dream work. Two colleges also had presidents who were very involved. 

• Active IR departments that produced accessible reports on students’ achieve- 
ment. Each school focused sharply on increasing IR capacity by hiring new IR staff 
and upgrading its data management system, which allowed it to better manage multiple 
IR demands. Each school also produced institutional- or department-specific reports on 
students’ success, with clearly delineated target goals for students’ achievement. 

• Regular evaluations of their interventions to improve student success. Each col- 
lege systematically tracked its interventions, often comparing students who were par- 
ticipating in a particular program with nonparticipants. These findings were dissemi- 
nated widely throughout the school, with a number of faculty and staff commenting on 
the results. 

• Strong faculty and staff leaders who played an active role in leading the colleges’ 
reform s and interventions. Each of the four colleges had strong participation from 
faculty and staff, who were involved in leadership and policymaking committees at 
their colleges and led the implementation of numerous strategies that were aimed at 
improving students’ success. 

• Integrated committee structures that allowed for regular communication among 
administrators, faculty, and staff. Each of the four schools developed numerous 
standing committees to monitor its reform efforts. These committees brought faculty, 
staff, and administrators together through regular communication, providing a clear 
and consistent voice in their colleges’ larger policy- and institutional decision-making. 

• Consistent attention to financial supports for strategy scale-up and numerous 
strategies operating at scale. Each of these college developed systematic ways to 
support broad-scale reforms, often by using federal or state grants to support the addi- 
tional costs. All four colleges had strategies that were reaching substantial proportions 
of their remedial students or students who were in college for the first time. 

• Heavy investment in professional development. Each of the four schools dedicated 
significant funds to training faculty and staff 
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tions to improve student success; and attention to scaling up program strategies that helped stu- 
dents be more successful. 

Some Culture of Evidence 

Ten Round 1 colleges (38 percent) had instituted many of the aspects of Achieving the 
Dream’s suggested improvement process in their schools, though not to the same degree as the 
colleges described above. Most schools had built (or started with) relatively capable institutional 
research departments, and most undertook longitudinal analyses of student outcomes and 
tracked the progress of at least some of their student success interventions. However, they also 
continued to have limitations in their data capacity that hindered their ability to undertake 
broader-based analyses. Other stakeholders, such as faculty and staff, also reported increased 
awareness of student achievement or had some exposure to reports on students’ success in par- 
ticular interventions. Faculty and staff were also generally involved in some aspects of the 
reform agenda at many of these schools, most often around implementing the colleges’ student 
success strategies, although their participation tended to be more limited than at colleges that 
had a better-developed culture of evidence. Finally, most colleges had some form of strategic 
planning process in place, which at times used student achievement data to inform decision- 
making, though the use of data was not consistent across these colleges. 

Weak Culture of Evidence 

As observed earlier, five colleges (19 percent) were still struggling to implement a 
number of Achieving the Dream’s recommended practices by spring 2009. Very weak institu- 
tional research departments were the primary difficulty hindering most of these schools’ ability 
to institute a broad data-driven culture. Most of these schools did not employ an institutional 
researcher continuously during their time in the initiative, relying instead on outside consultants 
or other, less well-trained staff to fill these positions. Sometimes these challenges were the re- 
sult of a poor economic situation, which kept the college from being able to attract good tech- 
nical support. Some schools also had difficulty implementing Achieving the Dream across mul- 
tiple campuses and providing sustained leadership to bring these disparate groups together. 



How Much Did Achieving the Dream Influence the Round 1 
Colleges? 

Many of the Round 1 colleges attributed their improvements to Achieving the Dream. 
As can be seen in Table ES.2, 19 out of 26 (73 percent) of the Round 1 colleges reported that 
Achieving the Dream had at least some influence on their development of a culture of evidence. 
However, they also noted other systems that were important in helping them develop a culture 
of evidence. For instance, a number of the schools discussed how their accreditation processes 
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Achieving the Dream: Community Colleges Count 
Table ES.2 



Influence of Achieving the Dream on the Culture of Evidence at Round 1 
Colleges and Progress Made from Spring 2006 to Spring 2009 



Number of Colleges 


Spring 2006 
Culture of 
Evidence^ 


Spring 2009 
Culture of 
Evidence 


Progress Made 
from Spring 2006 
to Spring 2009'’ 


Heavv influence: 8 colleges 


3 colleges 


Strong 


V ery strong 


H-1 


1 college 


Some 


Strong 


H-1 


3 colleges 


Weak 


Strong 


+2 


1 college 


Very weak 


Strong 


+3 


Some influence: 11 colleges 


2 colleges 


Strong 


V ery strong 


+1 


1 college 


Very weak 


Some 


+2 


6 colleges 


Weak 


Some 


+1 


2 colleges 


Very weak 


Weak 


+1 


Little influence: 7 colleges 


1 college 


Strong 


V ery strong 


+1 


1 college 


Weak 


Some 


+1 


2 colleges 


Some 


Some 


0 


1 college 


Very weak 


Very weak 


0 


2 colleges 


Some 


Weak 


-1 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of 
implementation research across all 26 Round 1 colleges, cross-referenced against reports submitted 
by college representatives. Achieving the Dream coaches, and data facilitators. 

NOTES: While some of the colleges in the "Little influence" category made progress in developing 
their cultures of evidence, they did not consider Achieving the Dream to be a primary motivator for 
these changes. 

““Spring 2006 culture of evidence ratings are based on implementation findings from MDRC and 
CCRC's baseline report on the Round 1 Achieving the Dream colleges (Brock et al., 2007, pp. 96- 
97). 

'’Progress ratings reflect changes in colleges' cultures of evidence between spring 2006 and spring 
2009, as measured by movement across the following rating scale: very weak, weak, some, strong, 
very strong. Eor example, a college that received a "very weak" rating in spring 2006 and a "very 
strong" rating in spring 2009 would receive a progress rating of "h- 4," indicating that it had 
progressed 4 levels in the rating scale. 
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or grants other than those from Achieving the Dream, such as federal Title V or TRIO 
grants, were the main drivers of the reforms at their colleges. Achieving the Dream was 
seen as “complementing” or “reinforcing” this agenda, but not necessarily as the primary 
inspiration for the colleges’ efforts to improve student achievement or to use data. Addi- 
tionally, a few colleges said that the motivation for their reforms stemmed from an internal 
push to increase student achievement, often from a strong leader who was already commit- 
ted to improving student success. 



What Program Strategies Were Developed for Students, and to 
What Effect? 

Colleges can expect real changes in student outcomes only when they extend meaning- 
fully improved programs and services to significant numbers of students. In other words, a 
strategy will make an observable impact on institutional performance only if it (1) improves 
academic outcomes among the students who are served, and (2) reaches enough students to 
“move the needle” on collegewide measures. 

Overall, the Round 1 colleges implemented a large number of strategies under the aus- 
pices of Achieving the Dream. Most of those strategies involved the implementation of direct 
programs and services that were designed to improve students’ college success. While the strat- 
egies ranged widely, from light-touch orientation activities to more intensive curricular reforms, 
several common themes emerged across the 26 colleges: 

• A large majority of strategies were designed to increase academic and social 
support systems for students, while only about one-fourth changed the con- 
tent and delivery of classroom instruction itself. 

• Nearly half of the colleges’ strategies targeted developmental education stu- 
dents, and one-third focused on students in their first year of college. Very 
few strategies targeted students based on their race, ethnicity, or economic 
status, though some colleges addressed achievement gaps indirectly through 
programs aimed at a broader group of students. 

• Despite colleges’ notable efforts to scale up their programs and services, the 
majority of strategies reached less than 10 percent of their intended target 
populations. Intensive strategies, such as curricular reforms or intensive ad- 
vising, which were defined by longer periods of contact time with students, 
were particularly unlikely to reach large numbers of students. Student suc- 
cess courses (courses aimed at introducing students to college life and en- 
hancing their study skills) were the sole high-intensity strategy to reach a 
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large number of students at a majority of colleges. This finding suggests that 
colleges often faced a trade-off between the intensity and scale of their inter- 
ventions. 

Most colleges had several additional strategies that did not reach students directly, but 
rather through (1) changes in college policies, and (2) collaborative relationships with external 
stakeholders, including local school districts and community members. 

Besides improving upon the programs, services, and policies that affect students. 
Achieving the Dream colleges are expected to leverage engagement among faculty and staff 
and improve their skills as part of the effort to increase student success. Indeed, the initiative 
highlights the importance of professional development in creating more systemic changes in 
community colleges’ improvement efforts. This proved to be an important area of college ac- 
tivity under Achieving the Dream, with every college reporting at least one professional de- 
velopment strategy and several reporting that this was a key focus of their work. Findings 
from an MDRC cost study of Achieving the Dream further highlight the centrality of this 
work: investments in professional development averaged $654,000 across the five Round 1 
colleges in the study, representing an important share of colleges’ spending on activities re- 
lated to the initiative.^ 

On average, each college implemented seven strategies, with all 26 colleges implement- 
ing over 200 strategies in total, representing an enormous investment of time and work by col- 
lege personnel. Also, even though, as already noted, most strategies reached less than 10 percent 
of their intended target, the colleges made noteworthy progress toward scale-up, as nearly all 
succeeded at expanding at least one direct strategy to reach at least 25 percent of its intended 
target population. However, a majority of strategies across the Round 1 colleges remained small 
in scale, particularly when they involved the kind of intensive contact that might be expected to 
meaningfully influence students’ performance. The result is that the benefits of promising inter- 
ventions were frequently extended to only a fraction of the students who were in need of more 
intensive assistance. 

It should not be surprising then, that the trends in student outcomes across the 
Round 1 colleges remained relatively unchanged across the pre-initiative and post-initiative 
periods. The analysis suggests that the average rates of persistence and graduation as well 
as the rates for completion of developmental math, developmental English, developmental 
reading, and gatekeeper math courses remained substantially the same throughout the pe- 



^Elizabeth Zachry and Erin Coghlan with Rashida Welbeck, Investing in Change: How Much Do Achiev- 
ing the Dream Colleges Spend — and from What Resources — to Become Data-Driven Institutions? (New 
York: MDRC, 2010). 
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riod, with modest improvements over time in the rates for completion of gatekeeper English 
and completion of coursework. 

In addition, few changes were seen in the outcomes of specific college and student 
subgroups. For instance, a separate analysis of the outcomes for several colleges that made 
the most progress in building a culture of evidence during their time in Achieving the 
Dream revealed no significant differences from the trends of the Round 1 colleges overall. 
Similarly, few changes were seen in the achievement gaps by race, ethnicity, and income. 
Though African American students showed some improvements in course completions and 
white students showed improvements in both course completions and gatekeeper English 
pass rates, those improvements over time were not large enough to close the achievement 
gap between the two groups. 

While these descriptive trends seem to suggest that Achieving the Dream had few posi- 
tive effects on student outcomes, the results must be interpreted with caution. While the lack of 
movement in student outcomes may occur largely because the interventions affected a small 
proportion of students in a cohort, the stability of student indicators over time may also reflect 
the short time span over which outcomes were analyzed. That is, the analysis of two-year out- 
comes for students may not provide a long enough window to adequately reflect the changes in 
institutional capacity and the implementation of specific strategies given the very low levels of 
student achievement that most colleges faced initially. 



Conclusion 

While community colleges are a prominent part of the national conversation today, few 
organizations were concentrating on these institutions before 2004, and even fewer were devot- 
ing substantial funds toward their improvement. Moreover, before Eumina Foundation for Edu- 
cation launched Achieving the Dream, few colleges or other postsecondary organizations fo- 
cused on community college students’ success; rather, they focused more often on providing 
open access to postsecondary education. 

In 2010, Achieving the Dream became an independent nonprofit entity, and it is under- 
going a marked shift in its leadership and organization as a result, seeking to involve an even 
wider range of community colleges in its work. In a change from the past, community colleges 
will have to pay to participate in Achieving the Dream, though it is expected that some institu- 
tions may find sponsors. Many of the core activities that Achieving the Dream provides — such 
as coaching, data facilitation, and annual Strategy Institutes — will also be subsidized so that 
colleges do not have to cover the full cost of participation. 
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As Achieving the Dream moves into this new phase, the initiative, like its participating 
institutions, may need to refine its approach to make a stronger impact on students’ success. For 
instance, outside of implementing colleges’ strategies, the faculty and staff at many of the 
Round 1 colleges tended to be less involved than administrators in the colleges’ overall im- 
provement work. Similarly, only a handful of colleges made attempts to engage their part-time 
faculty and staff, who most often lead the developmental and gatekeeper courses that are so im- 
portant to students’ ultimate success in college. Additionally, while most colleges had expanded 
at least one strategy, the majority of strategies at these schools remained small in scale, leaving 
large proportions of students relatively untouched by the colleges’ Achieving the Dream work. 

These issues represent important challenges to Achieving the Dream’s vision for im- 
proved student achievement; however, they should also be placed in the context of the overall 
struggles facing the U.S. education system as a whole. Elementary and secondary schools have 
long faced an uphill battle in their efforts to improve students’ achievement, with only a relative 
few achieving large-scale reforms that successfully increased students’ academic skills or grad- 
uation rates. Far more have been mired in many of the same struggles that Achieving the Dream 
colleges face, including students with major academic challenges, poor funding streams, and a 
lack of faculty or staff engagement in larger school reform efforts. Indeed, school systems 
working with adolescents, such as middle and high schools, tend to have the most struggles with 
large-scale reform, mirroring many of the key challenges that community colleges are facing. 

As Achieving the Dream continues to expand, it might look to revise its framework for 
action by drawing on some of the lessons from efforts by K-12 stakeholders to improve stu- 
dents’ outcomes. Some of these steps include a more rigorous focus on changing the practices 
that are most likely to affect students — namely, the interactions that occur between students 
and instructors in the classroom. As K-12 schools have learned, changing such practices can be 
difficult and often requires heavy engagement with faculty and staff, strong and continuous pro- 
fessional development, and a carefully researched plan detailing how new practices will affect 
students’ achievement. Given this scenario. Achieving the Dream might look toward more de- 
finitive ways to involve larger proportions of faculty and staff in the change process while also 
aiming to develop a larger evidence base about what practices work to increase community col- 
lege students’ achievement. Similarly, the initiative might also seek to incorporate classroom- 
based measures of learning, which more clearly document students’ attainment of particular 
skills and practices, in its model for institutional improvement. Such measures might help 
bridge the gap in Achieving the Dream’s theory of action, which currently focuses on broad 
institutional changes in student outcomes that may take many years to manifest. 

While larger changes in students’ achievement may not yet have been realized. Achiev- 
ing the Dream has begun an unprecedented movement toward helping colleges improve their 
student outcomes and develop systems to sustain those efforts. Bringing faculty and staff voices 
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more concretely into colleges’ reform work and focusing more directly on improvements to 
classroom instmction and services may reap benefits for the next stage of the initiative’s work. 
Given the successes they have already had, the initiative and its participating colleges stand 
poised to move forward with this agenda and make the changes needed to help more communi- 
ty college students accomplish their goals. 
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Chapter 1 

Introduction 



Across the United States, enrollments at community colleges are soaring. In fall 2009, 
approximately 8 million students took courses for credit — nearly a 17 percent increase from 
two years before.' The rising demand is a result of many factors, including the weak economy. 
In such an environment, community colleges may help job seekers be more competitive. 
Community colleges are also more affordable than most alternatives; average annual tuition and 
fees are less than half those at public four-year colleges and universities, and one-tenth of those 
at private four-year colleges and universities. Finally, community colleges tend to be welcoming 
and accessible institutions. Probably for all these reasons, much larger percentages of nontradi- 
tional, low-income, and minority students enroll in community colleges than in four-year 
colleges and universities.^ 

The figures above indicate that community colleges do a good job of helping students 
get a foothold in higher education. Unfortunately, after students enroll, far too many seem to 
slip. National studies reveal that close to half of all students who begin at community colleges 
with the intention of earning a certificate or degree do not achieve their goal and are not 
enrolled in any college or university after six years.^ And despite the economic and cultural 
diversity at community colleges, students from higher-income families are much more likely to 
persist and earn diplomas than are students from lower-income families; similarly, white or 
Asian students are more apt to finish than black or Hispanic students.'* There are many explana- 
tions for these patterns, including the fact that many community college students are not ready 
to perform college-level work when they enter and may be juggling the demands of work and 
family while trying to go to school. Nevertheless, a growing number of policymakers and 
educators are calling upon community colleges to do better. 

“Achieving the Dream: Community Colleges Count” is a national initiative that was de- 
signed to increase the academic success of community college students, with a special emphasis 
on low-income students and students of color. Its approach is to help community colleges build a 
“culture of evidence” by using student records and other data to examine how students are 
performing over time and to identify barriers to their academic progress. From this analysis. 



'American Association of Community Colleges (2009); Kente Phillipe (Associate Vice President for Re- 
search and Student Success, American Association of Community Colleges), personal communication, July 9, 
2010 . 

^Provasnik and Planty (2008). 

^National Center for Education Statistics (2002). 

''Berkner, He, and Cataldi (2002). 
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community colleges are expected to develop intervention strategies that are designed to improve 
student outcomes, conduct further research on student progress, and bring effective programs to 
scale. Participating colleges received multiyear grants and technical assistance to help them meet 
the initiative’s goals. Lumina Foundation for Education launched Achieving the Dream and is its 
principal funder, though a large group of philanthropies and other organizations have joined 
Lumina to support the initiative. As of December 2010, more than 130 community colleges in 24 
states and the District of Columbia are involved in Achieving the Dream (see Figure 1.1). 

This report examines the accomplishments and lessons from the first group of 26 com- 
munity colleges to join the initiative.^ The “Round 1” colleges, as they are known, are located in 
Florida, New Mexico, North Carolina, Texas, and Virginia (see Table 1.1). Drawing on a 
combination of field visits, surveys of college administrators and faculty, program documents, 
and student achievement data, the report answers the following major questions: 

• To what extent did the Achieving the Dream colleges succeed in building a 
culture of evidence? What factors facilitated or hindered their efforts? What 
influence did Achieving the Dream have in changing college practices? 

• What strategies did the colleges put in place to improve student outcomes? 

What was the scale of those strategies? What did colleges learn about the ef- 
fectiveness of their strategies through their own evaluations? 

• What were the trends in student achievement at participating colleges? Are 
those trends similar to or different from trends of a comparison group of col- 
leges that did not participate in Achieving the Dream? 

The report covers a five-year period, beginning in the 2004-2005 academic year (when 
the first group of colleges joined Achieving the Dream) and ending in the 2008-2009 academic 
year (at the conclusion of their grants from Lumina Foundation). In analyzing the work and 
progress of the Round 1 colleges, it is important to note that these schools began their work in 
Achieving the Dream as the initiative was still evolving. Indeed, throughout the past several years, 
the initiative has expanded, modified, and codified many of its practices and supports, with the 
Round 1 colleges receiving less intensive versions of these messages during the early phases of 
their implementation. As such, the progress of the Round 1 colleges offers a look at the influence 
of the initiative during its early development, while its practices and supports were still being 
solidified. Additionally, because the process of building a culture of evidence is envisioned as a 
long-term effort, a future report will examine trends in student achievement over a longer 



^There were originally 27 colleges in the initiative, but one college withdrew after the first year, leaving 26 
colleges in Round 1. Additionally, some multicampus districts, such as Alamo Community College District, 
are counted as one college. 
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Achieving the Dream: Community Colleges Count 



Figure 1.1 

States and Colleges Involved in Achieving the Dream, 2004 to 2010 




■ Round 1 Achieving the Dream states (joined in 
2004-2005) 

□ Achieving the Dream states that joined after 2005 

A Round 1 Achieving the Dream colleges (joined in 
2004-2005) 

• Colleges that joined Achieving the Dream after 2005 




NOTE: Multi-college districts are represented with one dot on this map. 
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Achieving the Dream: Community Colleges Count 
Table 1.1 



Round 1 Colleges, Academic Years 2004-2005 and 2008-2009 



State/College 


Location 


2004-2005 

Enrollment” 


2008-2009 

Enrollment” 


Florida 


Broward College 


Ft. Lauderdale 


22,540 


24,634 


Hillsborough Community College 


Tampa 


16,157 


18,321 


Tallahassee Community College 


Tallahassee 


9,819 


10,947 


Valencia Community College 


Orlando 


20,727 


26,230 


New Mexico 


Central New Mexico Community College 


Albuquerque 


14,955 


16,930 


New Mexico State University-Dona Ana 


Las Cruces 


3,656 


4,872 


Santa Fe Community College 


Santa Fe 


2,307 


2,132 


Southwestern Indian Polytechnic Institute 


Albuquerque 


699 


536 


University of New Mexico-Gallup Campus 


Gallup 


1,814 


1,757 


North Carolina 


Durham Technical Community College 


Durham 


4,028 


3,819 


Guilford Technical Community College 


Jamestown 


7,513 


9,969 


Martin Community College 


Williamston 


717 


410 


Wayne Community College 


Goldsboro 


2,911 


2,954 


Texas 

Alamo Community College District Central Office'’ 
Northwest Vista College 


San Antonio 


3,448 


12,866 


Palo Alto College 


San Antonio 


3,416 


8,622 


San Antonio College 


San Antonio 


9,027 


24,812 


St. Philips College 


San Antonio 


5,521 


12,060 


Brookhaven College 


Farmers Branch 


6,629 


6,818 


Coastal Bend College 


Beeville 


2,893 


2,438 


El Paso Community College 


El Paso 


17,084 


14,884 


Galveston College 


Galveston 


1,583 


1,306 


Houston Community College System 


Houston 


26,341 


32,362 


South Texas College 


McAllen 


11,478 


14,162 


Southwest Texas Junior College 


Uvalde 


3,488 


3,341 


Vir 2 inia 


Danville Community College 


Danville 


2,470 


2,503 


Mountain Empire Community College 


Big Stone Gap 


1,846 


1,932 


Patrick Henry Community College 


Martinsville 


2,242 


2,194 


Paul D. Camp Community College 


Franklin 


821 


915 


Tidewater Community College 


Norfolk 


15,078 


18,312 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS). 

NOTES: “Full-time equivalent (FTE) 12-month enrollment numbers are shown. 

’’Four colleges in the Alamo Community College District are particpating in Achieving the Dream. The 
district is the recipient of the Achieving the Dream grant. 
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time period, when colleges will have had more time to institutionalize reforms that began under 
Achieving the Dream. 

Overall, research on the implementation of Achieving the Dream at these colleges re- 
vealed that many institutions had made important strides in building a stronger culture of 
evidence during their five-year participation in the initiative. In particular, colleges had en- 
hanced their leadership commitment to student success, increased their research capacity, and 
developed a number of interventions that are aimed at improving their students’ achievement. 
By spring 2009, 81 percent of the Round 1 colleges had adopted a number of practices asso- 
ciated with a strong culture of evidence, with only 19 percent of colleges still facing major 
challenges undertaking this work. The research also revealed a number of areas for improve- 
ment for the initiative, including the need for an increased focus on faculty and staff engage- 
ment, particularly among adjunct or part-time faculty; further definition of the initiative’s efforts 
to reduce achievement gaps by race, ethnicity, and income level; and a more systematic effort to 
help colleges expand and institutionalize their student success efforts. 

The remainder of this chapter is organized as follows: The next two sections provide a 
brief history of Achieving the Dream and describe the model and underlying theory. Those 
sections are followed by a description of the characteristics of students attending the Round 1 
colleges at baseline (that is, before the initiative began). The penultimate section describes the 
research methods and data sources that were used for the Achieving the Dream evaluation. The 
chapter concludes with an outline for the remainder of the report. 



A Brief History of Achieving the Dream 

The mission of Lumina Foundation is to help people achieve their potential by expand- 
ing access to and success in education beyond high school. In an internal document prepared in 
2002 — just two years after the foundation was created — staff identified community colleges 
as a “high-need area” that was ripe for systemic reform and direct assistance.'’ The foundation 
convened a meeting with community college experts and stakeholders from all over the country 
to vet ideas and solicit feedback on a grantmaking strategy that would increase the attainment 
and academic success of community college students, including those who are first-generation 
students, students of color, low-income students, and adult learners. After requesting that 
interested parties submit a statement of their qualifications, Lumina selected a group of organi- 
zations to help it design and operate what became Achieving the Dream: Community Colleges 
Count (see Table 1.2). The initiative was formally launched in 2004, and MDC, Inc. — a 



'’Lumina Foundation for Education (2002). 
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Achieving the Dream: Community Colleges Count 
Table 1.2 



Founding Investor and Partners in the Achieving the Dream Initiative 



Organization 


Primary Responsibilities in Achieving the Dream 


American Association of Community Colleges 
www.aacc.nche.edu 


Helps recruit and select colleges, houses Achieving the 
Dream Web site, establishes Web-based data tools, co- 
hosts annual Strategy Institute, co-leads national policy 
work, and helps disseminate lessons from Achieving the 
Dream to the higher education community 


Community College Leadership Program, 
University of Texas 
www.utexas.edu/academic/cclp 


Hires coaches, manages coaching activities, co-leads 
Kick-off Institute for new colleges joining the initiative, 
and leads trustee institutes in select states 


Community College Research Center, 
Columbia University 
http://ccrc.tc.columbia.edu 


Conducts research on a variety of issues related to 
community college students’ success, leads knowledge 
development activities for the initiative, and works in 
partnership with MDRC on the evaluation 


Jobs for the Future 
www.jff.org 


Coordinates state policy work, develops strategies to 
ahgn state laws and administrative procedures with 
Achieving the Dream goals, selects and supports state 
lead organizations, develops portfolio of promising state 
policies, and co-leads national policy work 


Lumina Foundation for Education 
http://www.luminafoundation.org/ 


Made founding investment in Achieving the Dream and 
has invested a total of over $67 million into the initiative 


MDC, Inc. 

WWW .mdcinc .org 


Manages and coordinates the overall initiative, hires and 
manages data facilitators, oversees communications, 
helps implement strategies to build community support 
and connect Achieving the Dream with other initiatives, 
chairs cross-organizational structural inequity team, and 
co-leads annual Strategy and Kick-off institutes^ 


MDRC 

www.mdrc.org 


Leads the evaluation of the initiative, develops and 
conducts a comprehensive evaluation plan that assesses 
the implementation of the overall initiative and the 
impacts of specific programs at select colleges 


Public Agenda 
www.publicagenda.org 


Works with select colleges to increase public awareness 
of campus issues; creates tool kits and provides technical 
assistance to help colleges engage various internal and 
external stakeholders; conducts focus groups with 
faculty, students, and community residents 



NOTE: “MDC is currently providing consultation, program services, and administrative support to 
Achieving the Dream as it makes the transition from a demonstration initiative to a sustainable nonprofit 
entity. 
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nonprofit organization dedicated to expanding opportunity and advancing equity — was chosen 
to manage it. 

From the earliest conversations, it was clear that Lumina Foundation and the partners 
had big ambitions for Achieving the Dream. They spoke about fostering fundamental changes 
in the culture and operations of community colleges — changes that would lead to measureable 
and lasting improvements in student outcomes. They also spoke about changing the context in 
which community colleges operate — everything from the mles governing their accreditation 
and funding to public awareness and support for community colleges. Over a series of meetings, 
the partners developed an “Integrated Action Plan” that called for coordinated work on several 
fronts:’ 

• Inside community colleges, by increasing their capacity to gather and analyze 
data on the academic needs and progress of their students and to use those 
data to inform decisions on how to strengthen classroom teaching, student 
services, and other college programs to raise academic achievement 

• In public policy, by encouraging state lawmakers and higher education offi- 
cials to adopt policies that could help community colleges serve their stu- 
dents more effectively, and to build consensus on state reporting require- 
ments and measures of student success 

• In research, by supporting investigation into factors that contribute to or im- 
pede the success of community college students and on effective programs to 
boost achievement 

• Among the public at-large, by increasing awareness of the work that com- 
munity colleges do and engaging more people in supporting their mission 

• Among national organizations that represent, support, and study communi- 
ty colleges, by promoting regular meetings, information sharing, and colla- 
boration 

Each of the partner organizations was assigned specific roles, as shown in Table 1.2. 
For example, the American Association of Community Colleges had primary responsibility for 
planning annual conferences (“Strategy Institutes”) that involved all the partners and communi- 
ty colleges participating in Achieving the Dream. Similarly, Jobs for the Future took the lead on 
state policy work. The Integrated Action Plan helped to ensure that the partners understood each 



’MDC, Inc. (2008). 
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other’s roles and that their work was mutually reinforcing. Ultimately, all efforts were geared 
toward increasing the capacity of community colleges to improve student outcomes. 

From the beginning, Lumina Foundation and the partners envisioned Achieving the 
Dream as a national initiative that would involve many states and institutions. Rather than 
spread resources too thinly, however, Lumina and the partners agreed to concentrate on particu- 
lar regions of the country and to expand gradually. They also believed that it would be advanta- 
geous to create a “critical mass” of institutions within certain states that could work together on 
state budget issues and other legislative priorities. Florida, New Mexico, North Carolina, Texas, 
and Virginia were chosen for the first round of Achieving the Dream efforts, largely because 
those states were perceived as having favorable climates for policy change, including stable 
funding and high-level support for community colleges. The initiative was also particularly 
concerned about promoting an equity-based agenda and including institutions with high 
concentrations of low-income students, students of color, and nontraditional students. Using 
data compiled by the U.S. Department of Education, the partners identified community colleges 
in each of those states that served the largest proportion of students of color and/or low-income 
students.* A Request for Proposals (RFP) was sent out to those institutions to invite them to 
apply for grants. Of the 105 institutions that received RFPs, 60 applied and 27 were selected to 
participate.® One college subsequently withdrew, leaving 26 Round 1 institutions. 

The Round 1 colleges are diverse in size, location, and student characteristics. The larg- 
est institution is Houston Community College, which had a full-time equivalent (FTE) enroll- 
ment of over 32,000 students in 2008-2009. The smallest institution is Martin Community 
College in North Carolina, which had an ETE enrollment of 410 students. The colleges were 
located in large and midsize cities, suburbs, and small towns. White students made up a majori- 
ty or plurality of students at most institutions, but nearly all the colleges enrolled substantial 
numbers of African American, Hispanic, or Native American students.'° 

Over the next several years, the partners held tme to their vision of making Achieving 
the Dream a national initiative. As noted earlier, more than 130 institutions in 24 states and the 
District of Columbia have gotten involved, representing nearly 1 million students across the 
country. In addition, a group of 20 funders have joined Eumina Eoundation in supporting the 



*To be eligible for Achieving the Dream, Round 1 colleges had to have at least one-third of their students 
identify themselves as African American, Hispanic, or Native American, or at least half of their students had to 
be receiving federal Pell Grants (a proxy for low income). Ehgibihty criteria were modified for later rounds. 

^Representatives from the partner organizations and several independent reviewers scored the colleges’ 
proposals based on four criteria: (1) the strength of the personnel that the colleges proposed would lead the 
initiative on their campuses; (2) the colleges’ commitment to the goals of the initiative; (3) a description of at 
least three programs or strategies to increase student achievement; and (4) the colleges’ vision for how 
participation in the initiative would lead to improvements in student outcomes. 

*®For more detail on the characteristics of individual colleges when they entered Achieving the Dream, see 
Brock et al. (2007), Appendix A. 




work at individual colleges and partner organizations. Following another extensive planning 
process that began in 2008, Achieving the Dream became an independent nonprofit entity in 
2010. In a marked shift from the past, community colleges will have to pay to participate in 
Achieving the Dream, though it is expected that some institutions may find sponsors." Many of 
the core activities provided by Achieving the Dream, such as coaching, data facilitation, and 
annual Strategy Institutes, will also be subsidized so that colleges do not have to cover the full 
cost of participation. 

Achieving the Dream was at the forefront of national efforts to improve student out- 
comes in community colleges and to hold institutions accountable for their performance. While 
there is no community college equivalent of No Child Left Behind — the federal law that 
requires states to establish proficiency standards in reading and math for students in public 
primary and secondary schools, and to conduct annual testing — most state governments 
require community colleges to report on student retention, graduation, transfer to other colleges 
and universities, and sometimes job placement rates. Some states are also beginning to use the 
budget process to reward or penalize community colleges for institutional performance.'^ 

Regional accreditation agencies are also pushing community colleges in the direction of 
increased accountability. For example, the Southern Association of Colleges and Schools 
requires colleges to develop a Quality Enhancement Plan that identifies an area for improve- 
ment; establishes specific, measurable goals; and engages in an ongoing assessment of progress 
toward meeting the plan. Similarly, the North Central Association of Schools and Colleges asks 
colleges to identify and undertake three “action projects” for improvement as part of its Aca- 
demic Quality Improvement Program.'^ Many community colleges either want to or are 
required to participate in these efforts, and are hungry for guidance. Achieving the Dream 
provides these institutions with a framework, guidance from outside professionals, and a 
network of institutions with similar goals and relevant experience. 



The Achieving the Dream Model and Theory of Change 

Achieving the Dream aspires to nothing less than to change the way community colleg- 
es “do business.” By gathering and analyzing data on student performance, college administra- 
tors and faculty are expected to gain a better understanding of where students are having 
academic difficulty and what might be needed to make students more successful. From this 
research, colleges are expected to implement strategies to improve student performance, 
evaluate the effectiveness of those strategies, and scale up those strategies that lead to improve- 



"Such changes were instituted as part of Achieving the Dream’s new sustainability model, which is focus- 
ing on diversifying the funding base and becoming less dependent on philanthropy. 

'^Dougherty and Hong (2006). 

'^Biswas (2006). 
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ment in student outcomes — and refine or eliminate those that do not. As shown in Figure 1.2, 
the Achieving the Dream model involves five steps: 



• Step 1: Leadership commitment. The college’s president and senior execu- 
tives, with support from the board of trustees and faculty leaders, commit to 
making the changes in policy and resource allocation that are necessary to 
improve student outcomes. They also organize a team representing adminis- 
trators, faculty, and other members of the community to guide the process. 

• Step 2: Use of data to prioritize actions. The college mines student tran- 
scripts and other data to understand how students are performing over time 
— for example, whether students are progressing through developmental 
courses and into college-level courses, and whether students are returning in 
subsequent semesters. Colleges are also expected to analyze data by race or 
ethnicity, income, and other background characteristics to identify groups 
that may be in need of extra support or intervention. Colleges are encouraged 
to identify a manageable set of problems to ensure that they focus their re- 
sources to the greatest effect. 

• Step 3: Stakeholder engagement. The college engages faculty, staff, and 
other internal and external stakeholders in developing strategies for overcom- 
ing barriers to student achievement based on a diagnosis of the causes and a 
review of lessons gleaned from prior attempts to address such problems. 

• Step 4: Implementation, evaluation, and improvement of strategies. The 

college implements strategies for addressing priority problems, which may 
include such things as revamping college orientation and testing procedures, 
adopting new instmctional practices, or making improvements in student ad- 
vising programs. The college evaluates whether the strategies lead to im- 
provements in student outcomes and uses the results to make decisions about 
whether to continue with the strategies or make further improvements. 

• Step 5: Establishment of a culture of continuous improvement. The col- 
lege takes steps to institutionalize effective policies and practices. Particular 
attention is given to how resources can be used to sustain and expand proven 
strategies. Program review, planning, and budgeting are driven by evidence 
of what works best for students. Colleges are also encouraged to provide pro- 
fessional development for faculty and staff to advance their efforts to im- 
prove student achievement. 

In addition to these five steps. Achieving the Dream also placed an important emphasis 
on equity and overcoming achievement gaps, particularly among students of color and low- 
income students. The initiative initially made this component a cornerstone of its work by 
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Achieving the Dream: Community Colleges Count 



Figure 1.2 

Achieving the Dream’s Five-Step Process for Improving Student Success 
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STEPS 




STEPB 


Implement, 




Engage 


evaluate, 

improve. 




stakeholders. 




Improved student outcomes and reduced 

achievement gaps among subgroups of 

students based on five indicators: 

■ Completion of developmental courses and 
progression to credit-bearing courses 

■ Completion of gatekeeper courses, particularly 
first college-level or degree-credit courses in 
math and English 

■ Completion of attempted courses with 
a C or better 

■ Persistence from term to term and year to year 

■ Attainment of credentials 



SOURCE: Field Guide for Improving Student Success (2009), p. 1 1 , from the Achieving the Dream: Community 
Colleges Count initiative. Copyright 2009 by Lumina Foundation for Education. Reprinted with permission. Available 
online at www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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recruiting colleges with high proportions of low-income students or students of color. As 
Achieving the Dream grew and a more diverse group of institutions joined, the initiative 
continued to push this agenda by encouraging colleges to analyze the performance of these 
students and develop interventions aimed at increasing their success. 

To help colleges institute their reforms, Achieving the Dream provided both monetary 
and technical support to the participating institutions. The Round 1 colleges were awarded an 
initial year-long planning grant of $50,000 each, followed by annual grants of $100,000 for four 
years ($450,000 total).'"^ In addition, the colleges were aided by two consultants: a data facilita- 
tor, who helped them perform the data collection and analysis and interpret the results, and a 
coach, who helped them set priorities, build consensus, and implement strategies for improve- 
ment. These consultants spent 12 days each at their assigned colleges during the planning year 
and the first year of implementation, and then gradually reduced their time to three days each 
during the final year. Additionally, the initiative sponsored a kick-off conference and annual 
Strategy Institutes for all the Achieving the Dream colleges. Each institution sent teams of 
administrators and faculty to these events, where they learned more about the Achieving the 
Dream process, made plans for their own campuses, and shared ideas and lessons with other 
colleges on how to help students be more successful. 

The initiative also provided some supports that were aimed at helping colleges focus on 
achievement gaps between students by racial, ethnic, and income group, although this support 
was less concentrated than other efforts to improve colleges’ leadership and research capacity. 
For instance, equity was an important focus of the annual Strategy Institutes and was discussed 
in numerous publications documenting the initiative’s efforts. However, rather than provide 
coaching or facilitation to support colleges’ development of an equity-based agenda, the 
initiative tended to rely on the colleges’ own capacity for and interest in pursuing those efforts. 
As such, colleges were free to set their own priorities for their Achieving the Dream work, 
which may or may not have focused on particular racial, ethnic, or income achievement gaps.'^ 

Regardless of the goals they developed, all Achieving the Dream colleges were asked to 
try to “move the needle” on five measures of student success. These measures included (1) 
completion of developmental courses and progression to credit-bearing courses; (2) completion 
of so-called gatekeeper courses, including introductory college courses in English and math; (3) 
completion of attempted courses with a grade of “C” or better; (4) persistence from semester to 
semester and from year to year; and (5) attainment of college credentials. Colleges were 



Achieving the Dream demonstration colleges in later rounds sometimes received smaller amounts of 
money over a shorter time period, depending on the funders that supported them. The majority received the 
same amount as did the Round 1 institutions. 

'^Frorn 2008 through 2010, Achieving the Dream created more tools and supports to enhance colleges’ 
efforts to reduce achievement gaps, including an online portal called the Equity Resource Center as well other 
pubhcations and tools. 
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expected to track these outcomes for each fall cohort of entering students and to make compari- 
sons with past cohorts to determine whether the outcomes improved over time. As noted above, 
colleges were also expected to break down the results for subgroups of students defined by race 
or ethnicity and by income. Lumina and the partners theorized that colleges that followed the 
Achieving the Dream five-step process would detect quantifiable gains on the five measures and 
would reduce gaps in achievement between subgroups. 

Throughout the history of Achieving the Dream, there was some tension between those 
who believed that gains in student outcomes should come quickly and those who believed that 
achieving gains was a long-term process. The Integrated Action Plan was vague on this point, 
indicating that colleges should see “measurable improvement in success rates” after four years 
and should have “achieved their long-term targets for student success” after eight years.*'’ What 
was clear was that the five-step process was iterative, so any improvements in student outcomes 
depended on the colleges’ ability to master the initial steps. Even if colleges did all that was 
asked of them, it could still take time for improvements to show up in the data, given the 
longitudinal nature of the measures. The oft-used metaphor of turning a ship was invoked by 
some of the partners to describe the work of Achieving the Dream. On the one hand, they firmly 
believed that the initiative would set community colleges on a better course; on the other hand, 
they recognized that changes would likely occur only gradually and might not be fully apparent 
for several years’ time. 



Characteristics of Students at the Achieving the Dream Colleges 
at Baseline 

Table 1.3 shows averages across the Round 1 colleges on a variety of student charac- 
teristics in fall 2002, two years before Achieving the Dream was launched. As explained later in 
this report, the data capture first-time students who were enrolling full time or part time. The top 
panel shows that as a group, the Round 1 colleges served more women than men and roughly 
equal proportions of younger and older students. A little more than 40 percent were white, 28 
percent were Hispanic, and 17 percent were black. The standard deviation — a measure of 
spread around the average — and the minimum and maximum values show that there is a high 
degree of variability across the Round 1 institutions. While a number of institutions served 
primarily students of color, most served a majority of white students. 

The second panel of the table presents baseline institutional averages for the major out- 
comes that Achieving the Dream is tracking and makes clear why improvement is so desperate- 
ly needed. For example, the table shows that a sizeable number of matriculating students in fall 



*'’MDC, Inc. (2008), Part 1-B, p. 1. 



13 




Achieving the Dream: Community Colleges Count 
Table 1.3 



Average Characteristics of Entering Students at the 26 Round 1 Colleges 
in Achieving the Dream, Fall 2002 





Average for 


Standard 


Minimum 


Maximum 


Characteristics 


All Colleges 


Deviation 


Value 


Value 


Entering student characteristics 










Gender (%) 










Male 


40.4 


5.3 


24.2 


52.2 


Female 


59.6 


5.3 


47.8 


75.8 


Age (%) 










18-20 years 


50.5 


9.0 


30.6 


65.7 


21-25 years 


18.6 


4.2 


10.6 


25.4 


26-30 years 


9.3 


2.4 


4.9 


15.2 


3 1 years and above 


19.8 


6.4 


7.5 


37.7 


Average age (years) 


24.6 


1.7 


21.5 


28.7 


Race/ethnicity‘“ (%) 










Hispanic 


28.3 


29.7 


0.0 


94.8 


White 


42.0 


22.7 


0.0 


97.9 


Black 


17.1 


15.5 


0.0 


52.5 


Other 


12.6 


23.8 


0.5 


100.0 


Baseline indicators of the fall 2002 cohort (%) 










Referred to any developmental math course 


61.8 


15.2 


25.0 


81.3 


Referred to any developmental English course 


32.7 


15.8 


7.5 


64.2 


Referred to any developmental reading course 
Referred to one or more developmental courses 


33.0 


10.7 


7.8 


54.0 


in any subject 


66.8 


13.0 


32.9 


82.4 


Completed a developmental course (any subject) 
within first 2 years 


40.1 


9.5 


18.6 


61.1 


Successful completion of gatekeeper math 
course within first 2 years 
Successful completion of gatekeeper English 


15.1 


5.4 


6.8 


24.9 


course within first 2 years 


26.4 


8.6 


15.0 


52.1 


Completed 80 percent or more of attempted 
credits within first 2 years 


52.2 


15.3 


21.8 


87.3 


Enrolled in fall of second year 


47.9 


6.2 


32.6 


59.7 


Awarded an associate's degree or certificate 
within 2 years 


3.5 


3.1 


0.0 


12.3 


Sample size 








26 


(continued) 
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Table 1.3 (continued) 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for this table used institutional means from all available data for the sample members in 
the Round 1 2002 cohort of the "student general” files from JBL. While Alamo Community College District is 
considered one site for the Achieving the Dream initiative, it has 4 colleges that report data separately to the 
database, bringing the total number of data-reporting institutions from 26 to 29. Because 2 of these colleges 
(Central New Mexico Community College and New Mexico State University-Dona Ana) did not submit 2008 
data to JBL, they are not included in the analysis. Data for the fall 2002 cohort at Guilford Technical Community 
College were also not submitted, bringing the sample size for this table to 26. 

The standard deviation is a calculated measure of the tendency of data to be spread out around the mean. 
Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

“Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are only in the multiracial category. 
Southwestern Indian Polytechnic Institute reported all students as American Indian/Alaskan Native, which falls 
in the "Other" category. 



2002 required developmental (remedial) education in at least one subject, with over half 
requiring developmental coursework in math. While this average is alarmingly high, there is 
wide variation across the Round 1 colleges. At one extreme, 81 percent of students at one 
institution were referred to developmental math; at another institution, the figure was 25 
percent. On average, about 40 percent of students who were referred to a developmental course 
in any subject completed it within two years — a fairly low indicator of success. Once again, 
these values represent baseline measures that the colleges tried to improve upon in subsequent 
years. 



Gatekeeper math and English courses refer to the first college-level or credit-granting 
courses in these subjects. As shown near the bottom of the table, students taking gatekeeper 
courses also had fairly low average completion rates after two years. For example, only 15 
percent of students completed a gatekeeper math course within the first two years, and only 
about 26 percent completed a gatekeeper English course. Similarly, just under half of these 
students were still enrolled in college one year later, while fewer than 5 percent of students were 
awarded an associate’s degree or certificate during this same time period. 



Research Methods and Data Sources 

To answer the research questions posed at the beginning of this chapter, a research team 
comprising MDRC and Community College Research Center (CCRC) staff studied the Round 
1 colleges over a period of several years. The team visited each of the 26 colleges two times: 
once during the first year of implementation, in spring 2006; and again during the last year of 
implementation, either in fall 2008 or spring 2009. The visits lasted two to three days and 
included interviews with the college president, vice presidents and deans of instmction, institu- 
tional research staff, the Achieving the Dream coordinator, and faculty members involved in the 
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initiative. Researchers followed a standard protocol to make sure that similar questions were 
asked of people in comparable positions at each institution. 

In addition to the field visits, the research team conducted a survey of college ad- 
ministrators and faculty at 23 colleges in 2007, about midway through their involvement in 
Achieving the Dream. The survey contained questions about the respondents’ involvement 
in the initiative and how often and in what ways they used data to make decisions in their 
jobs. At smaller institutions, all administrators and faculty members received the survey; at 
larger institutions, a random sample was surveyed. The response rate to the survey averaged 
73 percent among administrators and 60 percent among faculty. A more detailed explana- 
tion of the survey methodology and the questions that were asked of respondents is included 
in Appendix A.*’ 

Not surprisingly, an initiative of the size and complexity of Achieving the Dream also 
generated considerable documentation. The research team reviewed annual reports that the 
colleges submitted to MDC, Inc., describing how they used their grants and detailing their 
accomplishments and challenges. These reports included information on the specific strategies 
that colleges developed to improve student outcomes on their campuses. The research team also 
reviewed annual reports that the coaches and data facilitators submitted on the work they 
performed with the colleges and their assessments of the colleges’ progress. 

Finally, the research team analyzed student records data that the colleges submitted to 
the Achieving the Dream database, managed by JBL Associates (a higher education consulting 
firm) under contract with CCRC. The database contains information on full- and part-time 
students entering the colleges each fall and allows researchers to track student performance over 
time on the measures listed above: completion of developmental courses and progression to 
credit-bearing courses, completion of gatekeeper courses, completion of attempted courses with 
a grade of “C” or better, persistence from semester to semester and year to year, and attainment 
of credentials. More information about those data and the methods used in this study is provided 
in Chapter 8. 



Organization of the Report 

As noted earlier, Lumina Foundation and the partner organizations that were responsi- 
ble for the initiative recognized that the process of building a culture of evidence would be 
ongoing and that improvements in student outcomes would likely take several years. For that 
reason, this report focuses mainly on implementation and is organized according to the five-step 
process described earlier. Chapter 2 delves into the issue of leadership: what Achieving the 
Dream hoped to accomplish, what assistance it offered, and what success it had in engaging 



*^See, also, Jenkins and Kerrigan (2008). 
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presidents and other senior officials at the colleges. Chapter 3 describes the colleges’ expe- 
riences in collecting and using data to diagnose problems and develop strategies for improve- 
ment. Chapter 4 explains the efforts that colleges made to engage faculty, staff, and external 
stakeholders in the initiative. Chapter 5 details the intervention strategies that the colleges 
developed to improve student outcomes. Chapter 6 describes the colleges’ efforts to institu- 
tionalize Achieving the Dream and establish a culture of continuous improvement. Chapter 7 
provides an overall perspective on colleges’ development of a culture of evidence and the 
influence of Achieving the Dream on their progress. 

An analysis of trends in student achievement is presented in Chapter 8. This analysis 
represents a “first look” at student outcomes and will be updated in a final report, which is 
scheduled for release in 2012. Chapter 9 concludes with reflections on challenges and successes 
in Achieving the Dream and provides recommendations for the continuous improvement of the 
initiative. 
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Chapter 2 

Leadership Commitment 



Of central importance to the implementation of any institutional reform is strong and 
committed leadership. Much of the literature on institutional change has posited that visionary 
leadership is critical for creating and maintaining momentum in large-scale reform efforts.' 
School leaders often set the tone for an institution’s work and hold critical leverage over the 
school’s development and growth. This is particularly true in community college systems, 
where presidents and chief executive officers have a relatively high level of authority over the 
everyday practices in their institutions.^ 

Achieving the Dream has long advocated that community colleges seeking to improve 
their students’ achievement focus first on developing their leaders’ commitment to student 
success. As noted in the Achieving the Dream “framing paper” detailing the conceptual frame- 
work of the initiative: 

Presidential leadership is essential to bring about institutional change that 
will improve student outcomes. The president or chancellor must have a vi- 
sion for student success and equity and must be able to mobilize broad sup- 
port for that vision throughout the college and community.^ 

As the initiative has developed, it has articulated the need for strong leadership com- 
mitment in even more concrete terms that are detailed in its key practices and theory of change. 
Leadership commitment now makes up Step 1 of Achieving the Dream’s five-step process for 
improving student success (see Figure 1.2 in Chapter 1), with colleges’ senior leadership urged 
to commit “to making changes in policy and resource allocation necessary to improve student 
outcomes.”'' In 2009, the initiative also published a number of practices that it had long advo- 
cated as essential components in colleges’ leadership commitment (see Box 2.1). These practic- 
es include bringing other leaders, such as colleges’ senior administrators, boards of trustees, and 
faculty and staff leaders, into the colleges’ institutional improvement work and gamering the 
commitment of those individuals to make “changes in policies, procedures, and resource 
allocations to improve student success.”^ 



'Evans (1996); Heifetz (1994); Senge (1990). 
^Tyree, Milliron, and de la Santo (2004). 
^MDC, Inc. (2006). 

''Achieving the Dream (2009), p. 10. 
^Achieving the Dream (2009), p. 15. 
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Box 2.1 



Principles and Practices for Increasing 
Student Success Through Institutional Improvement: 
Practices Associated with Principle 1, Committed Leadership 



Vision and Values 

• President/chancellor and leadership team support efforts to improve student learn- 
ing and college completion, not just increase enrollments. 

• President/chancellor and other senior leaders make an explicit policy commitment 
to improve performance for all students and achieve equity in student outcomes 
across racial/ethnic and income groups. This policy is communicated to faculty, 
staff, students, and the community. 

• Institutional communications, including news releases, the college’s Web site, 
appropriate print publications, and so forth, address the student success agenda. 

Commitment 

• College’s board expects, and the president/chancellor provides, regular reports on 
student outcomes and the impact of institutional efforts to improve success rates 
across all student groups. 

• Senior leaders demonstrate willingness to support changes in policies, procedures, 
and resource allocation to improve student success. 

• Faculty leaders actively support a culture of inquiry and evidence and a broad- 
based student success agenda. 



SOURCE; Field Guide for Improving Student Success (2009), from the Achieving the Dream: 
Community Colleges Count initiative. Available online at 

www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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In addition to advocating for a general student success agenda, senior leaders are ex- 
pected to make an explicit commitment to reducing achievement gaps across different racial, 
ethnic, and income groups. Since its inception. Achieving the Dream has placed a high 
priority on improving the outcomes of low-income students and students of color. At first, the 
initiative focused only on those colleges with large proportions of African American, Hispan- 
ic, Native American, or low-income students, and the Round 1 colleges were expected to 
commit to overcoming the achievement gaps between these students and the rest of the 
student body.'’ This commitment was expected to begin with the president and other senior 
leaders at the college, who were then charged with spreading that commitment to other 
faculty and staff at the school. However, as discussed below. Achieving the Dream’s efforts 
to help colleges focus on overcoming these achievement gaps varied, leading some colleges 
to direct their energies to other areas. 

Although Achieving the Dream has codified a number of its practices and supports, this 
report focuses on the Round 1 colleges, which began their work while Achieving the Dream 
was still under development. Thus, although Box 2.1 emphasizes a number of practices that 
colleges are expected to emulate, these practices were not presented in such an iterative form 
until the final years of the Round 1 colleges’ implementation efforts. Similarly, while coaching 
has been with the initiative since its early beginnings, other supports, such as Trustee Institutes 
and other leadership meetings, were under development while the Round 1 colleges were 
implementing their reforms. As such, this chapter offers a look at the initiative’ s early develop- 
ment, when its practices and supports were still being solidified. 

The sections that follow describe the extent to which the 26 Round 1 colleges devel- 
oped broad-based leadership commitment within their institutions. The implementation research 
revealed that presidents, senior administrators, and faculty leaders demonstrated considerable 
support for improving students’ success at these colleges, and that Achieving the Dream had 
some influence on increasing the commitment of these leaders. College presidents were general- 
ly aware and supportive of their colleges’ Achieving the Dream agendas, even if they tended to 
be less involved than other staff in the leadership of the initiative. College leaders also tended to 
be aware of the racial, ethnic, and income achievement gaps at their schools, although closing 
these gaps was not the main goal of most schools’ Achieving the Dream work. Finally, many 
leaders emphasized the important role that their Achieving the Dream coaches had played in 
helping them develop their student success agendas. 



'’Brock et al. (2007). 
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Levers for Change in Achieving the Dream 

To facilitate leaders’ commitment to student success, the initiative partners recommend 
that colleges develop several key leadership committees charged with moving their student 
success agendas forward. First, colleges are expected to develop a core team, which includes 
the president, chief academic officer, chief student services officer, institutional research (IR) 
director, and faculty leaders. This core team of leaders is expected to head up the college’s 
institutional change agenda and plan out tactics for increasing students’ success. Achieving the 
Dream also recommends that a second team, called the data team, be formed to engage the IR 
director, faculty, and staff in analyses of the college’s performance and student outcomes, with 
special attention to low-income students and students of color.^ 

Achieving the Dream also provides a number of supports to participating colleges to 
spur their efforts to develop leadership commitment. First and foremost among these supports 
are the “coaches” whom the initiative has employed to work with college leaders. Leadership 
coaches were allotted 12 days to work with the Round 1 colleges (one coach per college) during 
their first two years in Achieving the Dream, followed by nine, six, and three days of coaching 
in their final three years of implementation, respectively.® Coaches are expected to assist 
colleges’ core teams in three broad areas, including (1) advising colleges on how to address 
priority issues and manage the politics of change; (2) strengthening the teams’ communication 
and openness to using data to confront and solve difficult issues; and (3) fostering continual 
reflection on the teams’ and the colleges’ performance and effectiveness.® In preparation for this 
work, each Achieving the Dream coach receives an initial two-day training and attends a two- 
day coach retreat once per year. 

In addition to assigning coaches. Achieving the Dream provides several forums and re- 
sources to support college leaders’ commitment to improving students’ success. Upon joining 
Achieving the Dream, each college is expected to send senior leaders, including the college’s 
president, to a four-day Kick-off Institute to begin planning for the first year of work. The 
initiative also hosts an annual, week-long Strategy Institute, where all of the Achieving the 
Dream colleges gather and share their implementation plans and interventions aimed at increas- 
ing students’ success. During the Strategy Institute sessions, particular forums are directed 
toward senior leaders, including a three-hour President and CEO Roundtable, which allow for 
further communication among senior administrators across the colleges. Some college leaders 
may also participate in other Achieving the Dream events, such as a Trustee Institute held 



’MDC, Inc. (2008). 
®MDC, Inc. (2008). 
®MDC, Inc. (2008). 
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annually for the Gulf Coast (Texas) colleges, or ad-hoc trainings or institutes focused on issues 
such as strategy development or increasing institutional research capacity.'® 

Finally, Achieving the Dream saw improving the outcomes of low-income students 
and students of color as an important leadership effort. Through the encouragement of senior 
leaders, colleges were expected to disaggregate their student outcomes data in order to 
examine and compare the achievement of differing racial, ethnic, and income groups. Equity 
was also featured as a topic at the initiative’s yearly Strategy Institutes, and colleges were 
asked to report on any equity initiatives they developed. However, while this commitment 
existed, a racial-, ethnic-, or income-equity focus was not a required part of colleges’ Achiev- 
ing the Dream work, nor did the initiative develop specific supports, such as coaching or 
facilitation, to help colleges develop a more equity-focused agenda. Instead, colleges were 
encouraged to develop their own goals for their Achieving the Dream work and had the 
option to focus their efforts on other student groups. Additionally, the initiative and its 
coaches tended to encourage colleges to develop interventions aimed at large groups of 
students rather than boutique programs for certain groups. Therefore, while racial, ethnic, and 
income equity was an important part of Achieving the Dream, the initiative did not always 
make this agenda the centerpiece of its support efforts, choosing instead to depend on colleg- 
es’ own internal capacity to promote this work." 



Leadership Commitment to Student Success 

Early Commitment 

As noted in the 2007 baseline report on Achieving the Dream, by MDRC and the 
Community College Research Center (CCRC),'^ most Round 1 colleges began Achieving the 
Dream with strong leadership support for improving student outcomes. In spring 2006, all 
Round 1 college presidents vocalized a need to improve student outcomes and to eliminate 
achievement gaps across different student subgroups. Many also saw synergies between 
Achieving the Dream’s goal of building a culture of evidence and state, federal, and accredita- 
tion agencies’ focus on tracking student outcomes. However, while voicing strong support for 



'®In the summer of 2009, Achieving the Dream also produced a series of practical guides to help colleges 
plan better for implementing Achieving the Dream within their schools; however, these guides were released 
after the Round 1 colleges had completed their five-year demonstration phase in the initiative and thus were 
less relevant for this period of study. 

"As noted in Chapter 1, from 2008 through 2010, Achieving the Dream created more tools and supports 
to enhance colleges’ efforts to reduce achievement gaps, including an onhne portal called the Equity Resource 
Center as well other publications and tools. 

'^Brock et al. (2007). 
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the initiative’s goals, most presidents were not as involved in leading their colleges’ Achieving 
the Dream efforts as the initiative had hoped. Many presidents had delegated responsibility for 
implementing the initiative to other college personnel, with nearly half of the colleges appoint- 
ing senior administrators to lead this work while others designated faculty, staff, or outside 
consultants to lead the charge. Additionally, leadership turnover complicated the Achieving the 
Dream implementation process at about one-third of the colleges, making it difficult for a 
number of schools to quickly ramp up their plans. Therefore, while presidents tended to voice 
strong support for Achieving the Dream’s goals in the early years of implementation, their on- 
the-ground leadership of the initiative at their institutions often differed from Achieving the 
Dream’s recommended stmcture. 

Commitment After Five Years in Achieving the Dream 

Presidents, Senior Administrators, and Faculty Leaders 

Given leaders’ early commitment to student achievement in 2006, it is not surprising 
that many still voiced a strong commitment in spring 2009, when the Round 1 colleges were in 
their fourth and final year of implementing their Achieving the Dream reforms. As in their early 
years in the initiative, the majority of the Round 1 presidents, senior administrators, and faculty 
leaders expressed a strong commitment to improving student success (see Figure 2.1). Just over 
one-third of the presidents at these schools were strong drivers of the colleges’ overall student 
success agendas (not shown), with faculty and staff often noting that their presidents had a 
“passion” for student success or that they were “clearly driving the agenda” for improving 
students’ achievement. Tidewater Community College’s president exemplified these leaders’ 
strong commitment to student success: “I want us to be a benchmark for student success in 
much the same way that we are a benchmark in recmitment [and] entrepreneurialism. . . At the 
end of the day, all of the glitz in the world falls shallow. . .if it doesn’t have [student success] as 
its base.” 



A large majority of senior administrators and faculty leaders also voiced a strong com- 
mitment to improving students’ achievement, with many observing that Achieving the Dream 
had a positive influence on their commitment (see Figure 2.2). In the words of one student 
support services administrator. Achieving the Dream has allowed college personnel to access 
data in “real time” and made them “more focused” on improving students’ success. A coordina- 
tor at another school noted that Achieving the Dream “wasn’t about just one program or service. 
It was transformational and groundbreaking for the entire institution.” 



'^Brock et al. (2007). 
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Achieving the Dream: Community Colleges Count 
Figure 2.1 

Leadership Commitment to Improving Student Success Among Presidents, Senior 
Administrators, and Faculty Leaders in Achieving the Dream Colleges, Spring 2009 



■ Strong commitment 




Type of Staff 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 

Achieving the Dream’s influence on senior administrators and faculty leaders may have 
been in part a result of the high level of involvement these individuals had in their colleges’ 
Achieving the Dream agendas. Senior administrators and faculty leaders were moderately to 
highly involved in implementing their colleges’ Achieving the Dream plans at nearly all of the 
Round 1 colleges (see Figure 2.3). Higher- and lower-level administrators were most often the 
head of colleges’ core teams (69 percent of the Round 1 colleges) and leading colleges’ Achiev- 
ing the Dream work. While faculty led a few colleges’ core teams, they tended to be most often 
involved in the development and implementation of colleges’ strategies, rather than serving on 
colleges’ higher-level decision-making committees (discussed in Chapter 4). 
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Achieving the Dream: Community Colleges Count 
Figure 2.2 

Influence of Achieving the Dream on College Presidents’, Senior Administrators’, and 
Faculty Leaders’ Self-Reported Commitment to Improving Student Success, Spring 2009 




Type of Staff 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 



While senior administrators and faculty leaders played key roles in Achieving the 
Dream, fewer Round 1 presidents were actively involved in colleges’ Achieving the Dream 
efforts. Only four Round 1 presidents were leading their college’s core team, while most college 
presidents delegated this work to other administrators or faculty (see Figure 2.3). Similarly, 
presidents were split as to the influence of Achieving the Dream on their visions for student 
success. While nearly half said that Achieving the Dream played at least a moderate role in 
shaping their agendas, over half did not see the initiative as a primary motivator of their work 
(see Figure 2.2). The initiative’s lesser influence on these individuals was the result of several 
different factors, including presidents’ focus on other pressing issues at the college (such as 
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Achieving the Dream: Community Colleges Count 
Figure 2.3 

Involvement of Presidents, Senior Administrators, and Faculty Leaders in Student Success 
Reforms at the Round 1 Colleges in Achieving the Dream, Spring 2009 
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SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 
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precarious financial situations, accreditation process requirements, presidents’ own personal 
visions, and leadership turnover). 

Given college presidents’ busy schedules and the multiple agendas that inform their 
work. Achieving the Dream’s lower level of influence on these individuals may not be surpris- 
ing. Despite the fact that the initiative was not the primary driving force behind many presi- 
dents’ visions, most appreciated Achieving the Dream for the systematic focus it brought to 
student success. A majority of colleges noted that the initiative helped them develop a more 
focused message about student success and coordinate their efforts to improve students’ 
achievement. For instance, while one president remarked that her vision has “always been about 
student success,” she felt that Achieving the Dream had been “remarkable” in “focusing their 
efforts.” Similarly, a number of multicampus colleges noted that the initiative had helped them 
develop a clear, cross-campus message and had helped with coordinating the work across their 
different institutions. 

Board of Trustees 

The Achieving the Dream model holds that colleges’ boards of trustees, which are gen- 
erally responsible for helping colleges develop their mission and work, should regularly receive 
reports on students’ achievement and colleges’ efforts to improve their success. Most Round 1 
colleges’ boards demonstrated an awareness and support of the colleges’ efforts to improve 
student achievement, even if they did not necessarily know about the colleges’ participation in 
Achieving the Dream. Virtually all boards reviewed at least some reports on students’ success. 
The boards at five Round 1 colleges were particularly active and credited Achieving the Dream 
with their heightened awareness of students’ outcomes. For instance, at Tallahassee Community 
College, the board had “workshops before every board meeting... [to] focus on [their] strategic 
plan data and evidence on how we are doing.” Similarly, South Texas College provided regular 
updates to its board, which led the board to become more interested in performance data as 
opposed to enrollment or budgetary data. However, as might be expected, most colleges’ boards 
were far less knowledgeable about the goals of Achieving the Dream or how their colleges’ 
student success strategies were directly connected to their Achieving the Dream work. 



Leadership Commitment to Reducing Achievement Gaps 

As noted previously, a focus on improving equity across racial, ethnic, and income 
groups was an important focus of Achieving the Dream from its inception. In line with this 
focus, most of the leaders at the Round 1 colleges tended to have an awareness of achievement 
gaps at their schools. For instance, a large majority of the Round 1 colleges disaggregated their 
student outcomes data by race, ethnicity, or income (or some combination of all three); only 
four colleges did not analyze these subgroups or stated that analyses by race, ethnicity, or 
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income were less important than looking at other student subgroups. Many college leaders 
noted that an analysis of these disaggregated data had increased their overall awareness of 
achievement gaps, and many credited Achieving the Dream for this heightened sensitivity. 

While most colleges disaggregated their data for different racial and income groups, 
these analyses did not always translate into a specific focus on achievement gaps. Some leaders 
acknowledged that race or income was a big issue for their campus, and just over half of the 
colleges developed at least one strategy that was aimed at improving the success of these 
students. As described in Box 2.2, the leaders at some colleges, such as Valencia Community 
College and Durham Technical Community College, saw promising improvements in the 
achievement gaps among these students. Several of these leaders were quite passionate about 
this agenda and strongly supported Achieving the Dream’s focus on these issues. As one 
college president mentioned, “I get embarrassed when my friends talk about [not having] the 
courage to have those [race] discussions. That is what we signed up to do to help students... 
[Race] is the fault line in America. Lumina provides a huge cover for us to be able to [address 
racial disparities] ... I hope they never walk away from that focus.” 

However, despite these promising actions, overcoming racial, ethnic, and income 
achievement gaps was not a key goal at the majority of Round 1 colleges. Only eight leaders 
made explicit attempts to raise awareness about those issues. Most of these schools focused on 
either race and ethnicity or income (rather than both), with the majority of schools focusing on 
the gaps between students of color and white students. The colleges that did not make race, 
ethnicity, or income equity issues their main focus gave several reasons for their choice. First, 
the population at some of the schools comprised mostly students of color or low-income 
students, making it less meaningful to separate out these students from the general student 
population. Additionally, some schools found few achievement gaps along different racial, 
ethnic, or income lines, and one school found that students of color were outperforming their 
white peers. A majority of colleges also explained that they were more interested in helping all 
students succeed rather than just one specific subset, arguing that nearly all of their students 
were disadvantaged in some way. A couple of these schools purposely shied away from an 
explicit focus on race, noting that such conversations were too contentious for their college 
communities. Others chose to focus their efforts on other student subgroups, such as develop- 
mental education students, whose need for additional services was, they felt, greater. Finally, a 
couple of colleges noted some confusion about Achieving the Dream’s equity agenda, with one 
college noting that its coach and data facilitator had pointed the college away from equity- 
focused strategies in favor of those aimed at larger groups of students. 

Other outside factors may have influenced colleges’ timid response to Achieving the 
Dream’s equity agenda. First, in the early part of the twenty-first century, several high-profile 
cases were being tried in the U.S. Supreme Court, which brought into question the legality of 
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Box 2.2 



Spearheading a Focus on Equity: Case Studies of Four Coiieges 

While few colleges developed overarching mechanisms for addressing racial, ethnic, and 
income achievement gaps, many schools designed select strategies to improve the perfor- 
mance of their low-income students or students of color. The stories helow represent sev- 
eral different models upon which colleges based their pursuit of this goal. 

Valencia Community College: Early Successes in Closing Gaps 

Honored as the first recipient of the Leah Meyer Austin Institutional Student Success 
Leader Award,“ Valencia made promising strides from 2004 to 2008 in narrowing the 
achievement gap between its African American and Caucasian students (from 13 percent 
to 4 percent) in gatekeeper courses, as well as reversing the gap between Hispanic and 
Caucasian students in gateway courses. 

When reviewing internal data on student enrollment and achievement, Valencia noted that 
students of color, developmental education students, and students who were taking math 
courses were performing poorly. In order to address these challenges, the college chose to 
focus its intervention strategies on six gateway courses, including four mathematics 
courses, an English composition course, and a United States government course. The col- 
lege used a combination of supplemental learning, learning communities, and a mandatory 
student life skills course as strategies to close gaps in student achievement in its gateway 
courses. 

As a result of this work, Valencia stands out among the Round 1 colleges for creating a 
systematic, collegewide approach to improving equity under Achieving the Dream. By 
2009, more than 29,000 Valencia students had taken at least one of the six gateway 
courses, and students’ performance improved in each of these courses. 

Durham Technical Community College: “Visions” Minority Male Leadership 
Initiative 

After discovering that persistence and graduation rates among its African American male 
students were lower than those of other student groups, Durham Tech decided to imple- 
ment a minority male leadership initiative to help these students overcome their obstacles 
and succeed in college. 

This initiative, entitled the “Visions” program, provided a variety of supports and engage- 
ment activities for men with lower-level academic skills. This multifaceted approach in- 
cluded mentoring, intensive academic advising in a weekly discussion group, artistic de- 
velopment through a jazz ensemble, financial support through scholarships and job 
placement services, leadership opportunities to serve as tutors for children at a local ele- 
mentary school, and expanded academic opportunities through visits to local universities. 

(continued) 
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Box 2.2 (continued) 

Durham Tech observed promising increases in Visions students’ persistence rates, 
which were higher than the average persistence rate of other African American male stu- 
dents. With this program at the forefront, Durham Tech has made equity and overcoming 
achievement gaps a central focus of its Achieving the Dream work. For example, the col- 
lege has completed a stmctural inequity audit to identify and address any barriers to suc- 
cess that its existing policies and procedures might create for low-income students and 
students of color. 

Tallahassee Community College: Black Male Achievers Program 

Tallahassee Community College also noted gaps in the persistence, attendance, and grades 
of its African American male students. It responded by establishing a Black Male Achiev- 
ers program, which offered a variety of academic and student services to help African 
American males stay in college and graduate. The program has also promoted personal 
enrichment and social activities, such as visits from the Tuskegee Airmen and service 
projects in the community, to advance students’ intellectual growth and social connections. 
This strategy represents an important step in Tallahassee’s effort to explicitly and directly 
engage its campus on the topic of reducing achievement gaps across racial groups. 

Santa Fe Community College: Diversity Initiative 

As a traditionally Hispanic-serving institution, Santa Fe Community College became more 
aware of its students’ needs after disaggregating student outcomes data as part of its work 
under Achieving the Dream. To propel the college’s commitment to equity, the president 
has spearheaded a diversity initiative focused on tackling Fhspanic and Native American 
students’ performance issues and hiring staff that better represent the college’s student 
population. The diversity initiative aims to refine the college’s mission, vision, and core 
values to reflect a commitment to equity and student success. 



“The Leah Meyer Austin Award, sponsored by Lumina Foundation for Education and administered 
by the American Association of Community Colleges, recognizes outstanding institutional 
achievement in creating excellence and equity through (1) committed leadership; (2) documented 
student success strategies; (3) broad engagement; (4) use of evidence to improve programs, pohcies, 
and practices; and (5) systemic institutional improvement. 



using race in college admissions decisions. ''' Such actions arguably created a less welcoming 
environment for a focus on particular racial and ethnic groups, leading some college leaders to 
move away from such an agenda.'^ Additionally, as noted above, though Achieving the Dream 
encouraged an equity-based focus, it did not provide substantial support for helping colleges 



'^Gruffer versus Bollinger (2003); Gratz vi. Bollinger (2003). 
'^Arenson (2003); Associated Press (2008). 
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undertake such work. The Round 1 colleges’ mixed response suggests that a much more 
intensive focus, with more external supports, may need to be placed on this equity agenda for 
colleges to undertake this work. 



Senior Leaders’ Exposure to Student Outcomes Data 

One of the key tenets of Achieving the Dream is that senior leaders at the college 
critically review data on their students’ achievement and use these findings to develop and 
revise their plans for increasing students’ success. In order to assess administrators’ and 
faculty members’ use of data, MDRC and CCRC researchers administered a survey to 23 of 
the 26 Round 1 colleges.'^ (See Appendix A for a discussion of the methodology.) The 
findings from this research revealed that, overall, administrators at a majority of Round 1 
colleges tended to review institutional-level student outcomes data, such as information on 
retention and graduation rates, at least once per year (see Figure 2.4). These leaders typical- 
ly reviewed other types of data as well, including classroom evaluations and grades (Figure 
2.4), outside research including qualitative data, and disaggregated data for different student 
subgroups (see Figure 2.5). 

While administrators reviewed multiple types of data, they did not always analyze 
these data on a regular basis. With the exception of enrollment data, most administrators at 
the Round 1 colleges reported that they examined data related to student outcomes and 
research less than four times per year. Additionally, findings from the survey indicate that 
the review of these data varied significantly by department, with institutional research and 
academic affairs administrators analyzing these data more frequently than other depart- 
ments. Administrators who had a faculty role were also more likely to examine data related 
to student outcomes or data disaggregated by race, ethnicity, or income, and were more 
likely to engage in organized discussions about data relating to student achievement gaps. 
Thus, as might be expected, administrators’ review and use of data tended to differ depend- 
ing on the role they played at the college.'* 



'^Tidewater Community College, Valencia Community College, and Southwestern Indian Polytechnic 
Institute did not participate in the faculty and administrator survey. For more information on faculty’s use of 
data, see Chapter 4. 

'^For more details on the Achieving the Dream faculty and administrator survey, see Jenkins and Kerrigan 
(2008). 

'*For more information, see Jenkins and Kerrigan (2008). 
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Achieving the Dream: Community Colleges Count 



Figure 2.4 

Frequency with Which Administrators Used Student Outcomes Data, 
by Type of Data, 2007 to 2008 
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SOURCE: CCRC calculations using survey data collected from faculty and administrators at the Achieving the 
Dream Round 1 colleges. 

NOTE: Eaculty and administrator survey data were collected from September 2007 through Eebruary 2008, 
approximately one year before implementation research at the Round 1 colleges. The survey was disseminated 
at 23 of the 26 colleges. 
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Achieving the Dream: Community Colleges Count 



Figure 2.5 

Frequency with Which Administrators Used Data Disaggregated by Race and Income, 

2007 to 2008 
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SOURCE: CCRC calculations using survey data collected from faculty and administrators at the Achieving the 
Dream Round 1 colleges. 

NOTE: Eaculty and administrator survey data were collected from September 2007 through Eebruary 2008, 
approximately one year before implementation research at the Round 1 colleges. The survey was disseminated 
at 23 of the 26 colleges. 



Usefulness of Achieving the Dream’s Inputs 

A large majority of presidents and senior leaders noted that their Achieving the Dream 
coach was an important guide in helping them develop and advance their student success 
agenda. While many colleges reported that their coach was a good “cheerleader” for their cause, 
colleges found coaches most helpful when they provided concrete advice on particular aspects 
of their institutional improvement plans, such as bringing in new research or ideas for develop- 
ment, helping leaders prioritize program strategies, or providing guidance on how to engage 
faculty and staff in the colleges’ work. A number of colleges, particularly those that stmggled 
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with their improvement plans, also noted that their coach was very important in helping them 
“keep on track,” confront challenging issues at their schools, and frame their message to the 
college community and beyond. Finally, a few colleges noted that it was helpful to have an 
outside perspective on their work and that their coach helped bring legitimacy to their focus on 
student success. 

While few schools had any problems with their coach, those negative sentiments that 
did arise tended to center around colleges’ desire for more guidance or more information about 
Achieving the Dream’s expectations for their work. A handful of colleges noted that they 
wished their coach had spent more time “in the trenches,” including demonstrating a better 
understanding of their own specific community college environment. While a few of these 
colleges requested and received new coaches whom they felt were a better fit for their schools, a 
couple of colleges continued to struggle with their coaches throughout the duration of the 
initiative. Most of the colleges that had trouble with their coaches tended to be smaller, mral 
colleges, who felt that their coaches did not have a great understanding of the challenges facing 
their communities. 

A few other Achieving the Dream resources were identified as important supports for 
leadership commitment. A couple of colleges observed that their leaders had attended a Trustee 
Institute, which had been very useful in helping board members and tmstees become more 
aware of Achieving the Dream’s goals and mission. Others noted the utility of the Strategy 
Institute and some state-level meetings in helping them connect with and learn from other like- 
minded leaders. In general, however, few colleges discussed other Achieving the Dream inputs 
that helped shape their leaders’ commitment to or interest in the initiative or student success. 



Summary 

In general, the leaders of the Round 1 colleges tended to be committed to improving the 
success of their students and engaged in reviewing data on their students’ achievement. While 
most colleges did not develop a forthright agenda centered upon reducing achievement gaps 
along racial, ethnic, and income-level lines, many of these schools were aware of the challenges 
these students faced, often as a result of their participation in Achieving the Dream. Additional- 
ly, many colleges were sharing reports on student outcomes with their boards and actively 
engaging in plans for improving their students’ achievement. 

While many colleges had well-defined leadership throughout Achieving the Dream, one 
challenge that colleges continued to face was turnover among their presidential and senior 
administrator positions. Just under half of the Round 1 colleges continued to experience change 
in their high-level leadership, with 12 of the 26 Round 1 colleges hiring a new president from 
spring 2006 to spring 2009. Presidential turnover was an issue even at schools with leaders who 
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had a strong commitment to student success, although those schools had been able to maintain 
their focus through strong leadership from senior administrators or by hiring highly committed 
new presidents. However, at some colleges, leadership turnover was a debilitating event, which 
set back their student success agendas for months or years. 

Given these challenges. Achieving the Dream might consider ways to support a more 
diversified set of leaders at each college who can be distributed through a number of different 
networks when colleges inevitably experience leadership turnover. Active involvement from a 
number of senior administrators other than the president also seems to be an important attribute 
for colleges’ implementation of Achieving the Dream, as those schools that did not have senior 
leaders involved tended to have difficulty implementing their reforms. Therefore, having one or 
more senior leaders who are continuously involved in the implementation of the college’s 
improvement plans also appears to be important in helping colleges get their student success 
plans off the ground. 
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Chapter 3 

Use of Data to Prioritize Actions 



While leadership commitment to student success is key in the Achieving the Dream 
model, colleges’ development of their data capacity is the backbone of the initiative’s push 
toward creating evidence-based programs and policies. Traditionally, community colleges have 
tended to focus more on meeting the reporting requirements of government agencies and private 
funders and less on capturing the internal trends in students’ achievement.' Additionally, 
community colleges tend to have few staff to handle such tasks, making it challenging to focus 
on other types of data analysis.'' As a result, the institutional research (IR) departments at these 
institutions have generally been less focused on the student success-oriented data analyses 
promoted by Achieving the Dream. 

Given their limited data capacity, many colleges that join Achieving the Dream face a 
steep hurdle in gearing up their research departments to handle the types of data that the 
initiative finds most important. The ability of colleges to collect and analyze student and 
program data, as well as to use those data to inform institutional decisions, are of key impor- 
tance to Achieving the Dream’s vision of evidence-based decision-making. Without strong data 
systems and skilled research personnel, colleges may be unable to identify student outcomes of 
interest, to analyze where gaps exist in student achievement, or to track the progress of their 
programs. As a result, they may be unable to disseminate information about their overall 
performance or to make evidence-based decisions about their priorities for reform. Therefore, at 
many schools. Achieving the Dream’s call for a strengthened culture of evidence entailed a 
substantial expansion in their IR staffing and technology. 

As Step 2 of Achieving the Dream’s institutional improvement model, colleges’ use of 
data to prioritize actions is based on several key premises. First, colleges are expected to 
analyze longitudinal data on students’ achievement, as well as other evidence of students’ 
success, and to identify particular gaps in students’ outcomes. The initiative expects that most 
colleges will discover that students are performing below their expectations, and that these 
findings will motivate college faculty and staff to develop solutions for their improvement. 
Additionally, to make sure that they are targeting their resources appropriately, colleges are 
encouraged to evaluate a limited set of priorities and strategies aimed at increasing students’ 
success. 



'Merest and Jenkins (2007). 
^Merest and Jenkins (2007). 
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Such work also requires that colleges have the necessary staffing and technological ca- 
pacity. As detailed by the initiative in Principle 2 of its “Principles and Practices for Increasing 
Student Success Through Institutional Improvement” (see Box 3.1), colleges must develop 
stable informational technology (IT) systems to manage information on student outcomes and 
support the tracking of students’ longitudinal progress across multiple semesters and years. 
Additionally, they must have capable IR staff with the skills to analyze a number of indicators 
of student success. Achieving the Dream also recommends that colleges routinely disaggregate 
their data for students of different races, ethnicities, and income levels as well as other student 
characteristics to see what achievement gaps may exist among these groups. Furthermore, 
colleges are encouraged to collect qualitative data on their students through methods such as 
surveys and focus groups. Finally, Achieving the Dream expects that colleges will evaluate their 
strategies and interventions, using the results of their analyses to improve programming and 
policy. 



The initiative has now documented an extensive list of practices for participating col- 
leges, some of which are described in Box 3.1; however, it is important to note that the Round 1 
colleges began their work in Achieving the Dream while the initiative was still articulating this 
vision. While longitudinal data analysis and the disaggregation of data for different student 
subgroups have been priorities of the initiative since the beginning, the full codification of these 
practices evolved over the course of the initiative’s development, after the Round 1 colleges had 
already begun implementing their reforms. As a result, the Round 1 colleges had fewer re- 
sources when they began their work than many of the colleges that joined the initiative later. 

This chapter thus provides a look at how an early cohort of colleges progressed in de- 
veloping their data capacity while the initiative was still building its own infrastmcture. Overall, 
the implementation research found that most of the 26 Round 1 colleges made at least some 
progress in building their data capacity by increasing their IR staff, providing more data 
training, or bolstering their IT infrastructures. Most colleges had also made at least some 
progress in collecting diverse types of data, including longitudinal data on student outcomes, 
data disaggregated by different subgroups, and qualitative or descriptive data from surveys and 
focus groups. Despite these advances, many colleges had yet to fully incorporate data analyses 
and findings into their decision-making about strategy development or institutional policy 
changes. While some colleges reported that the supports provided by Achieving the Dream, 
such as data facilitators, helped encourage their progress, others provided mixed reviews of 
those services. 
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Box 3.1 



Principles and Practices for Increasing Student Success Through 
Institutional Improvement: Practices Associated with Principle 2, Use of 
Evidence to Improve Policies, Programs, and Services 

Information Technology (IT) and Institutional Research (IR) Capacity 

• IT capacity is adequate to meet the demand for data and institutional research. 

• Policies and procedures are in place to ensure the integrity of the data collected. 

• IR staff capacity is adequate to meet demand for data analysis and research. 

• IR staff effectively educates and assists college personnel to use data and research to 
improve programs and services. 



Process for Identifying Achievement Gaps 

• College routinely collects, analyzes, and reports longitudinal data on cohorts of stu- 
dents to track student progression and outcomes. 

• College routinely disaggregates student cohort data hy age, race, gender, income, and 
other factors to identify gaps in achievement among students. 

• College regularly conducts surveys and focus groups with students, faculty, and staff 
to identify strengths and weaknesses in programs and services, as well as opportunities 
for improvement. 



Process for Formulating and Evaluating Solutions 

• College routinely engages faculty, staff, and others across the campus community to 
review data on student achievement and help develop and refine strategies for address- 
ing priority prohlems. 

• College routinely evaluates the effectiveness of efforts to improve student success and 
uses the results to improve policy and practice. 



SOURCE; Field Guide for Improving Student Success (2009), from the Achieving the Dream: Com- 
munity Colleges Count initiative. Available online at 

www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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Levers for Change in Achieving the Dream 

In order to help colleges strengthen their evidence-based decision-making, Achieving 
the Dream has instituted a number of supports aimed at increasing colleges’ capacity to 
collect, analyze, and interpret data. First, the initiative employs a data facilitator who has 
expertise in institutional research and helps guide colleges in developing new systems for data 
usage. The data facilitator helps colleges decide what questions they should be asking about 
student achievement, what data they should collect to answer those questions, and how they 
should present those data to various stakeholders at the college. As explained in Chapter 1, 
each data facilitator is allotted 12 days to work with the colleges during their first two years in 
Achieving the Dream, followed by nine, six, and three days in each of the following years, 
respectively. Each data facilitator also initially attends a two-day training, as well as a two- 
day retreat each year.^ 

In addition to data facilitators. Achieving the Dream has created a national database to 
track students’ longitudinal progress at participating colleges. Each year, colleges are asked to 
submit individual-level data on their entering students’ achievement around Achieving the 
Dream’s five key indicators of student success,'* as well as pertinent information about students’ 
characteristics, such as race, ethnicity, gender, and income status.^ These student cohorts are 
then tracked over subsequent years through colleges’ continued data submission. Each college 
receives a detailed data submission manual, which provides guidance on how to set up its data 
files. In addition, colleges receive assistance from JBE Associates, the data management 
organization that manages the database. 

JBE Associates subsequently cleans and checks colleges’ data and provides individua- 
lized reports to each participating institution. These reports summarize students’ progress along 
measures such as the number of credits attempted and completed; referrals to and success in 
developmental reading, English, and math; persistence; and credential attainment. These reports 
also break down student outcomes for different student subgroups, as defined by race, ethnicity, 
income, age. Pell Grant eligibility, and gender. Since its beginning. Achieving the Dream has 
upgraded this system so that colleges can access this information through the initiative’s Web 
site and can build reports of their own with a new data management tool. In addition, JBE 



^MDC, Inc. (2008). 

'^As described in Chapter 1, the five key indicators are completion of developmental education courses and 
progression to credit-bearing courses; completion of gatekeeper courses, particularly first college-level or 
degree-credit courses in math and English; completion of attempted courses with a grade of “C” or better; 
persistence from term to term and year to year; and attainment of credentials. 

^Income status is inferred from students’ eligibihty for Pell Grants, a federal need-based aid program for 
low-income students. 
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Associates produces a bimonthly report, which summarizes trends in student outcomes based on 
an overall analysis of the data provided by all Achieving the Dream colleges. 



Colleges’ Data Capacity 

Early Data Capacity 

As might be expected from the challenges noted above, a number of Round 1 colleges 
were struggling to undertake the data analyses recommended by Achieving the Dream during 
their first year of implementation. In spring 2006, only 8 of the original 27 colleges (30 percent) 
were considered to have the data capacity needed to develop a culture of evidence.® At that time, 
most of the colleges described their resources for conducting institutional research as limited, 
with IR staff at some colleges working hard just to fulfill their basic federal and state com- 
pliance reporting requirements. More than a third of the colleges reported having too few IR 
staff to meet their institution’s demand for the higher level of research required to support 
broad-based, data-informed decision-making. Moreover, many of the colleges were considered 
to have antiquated or barely operational IT systems at that time, and nearly half reported having 
insufficient IT staff available to respond to their data requests. Finally, only half of the Round 1 
colleges were tracking students’ progress longitudinally, while only a handful were using data 
to guide their program efforts.’ 

Given such challenges, it is perhaps not surprising that so few of the Round 1 colleges 
were using data to guide their program and strategy development efforts. Instead, most colleges 
reported that their decisions to develop or expand programs and services were based primarily 
on administrator experience or intuition, even if their institutions had well-developed IR 
departments and data that were readily available.* 

Data Capacity and Use of Data After Five Years in Achieving the Dream 

Colleges’ Respect for Data Use 

Early in the initiative, a number of faculty and staff at the Round 1 colleges shied away 
from their schools’ data analysis, expressing concerns that data would be used to scrutinize their 



^Originally, 27 Round 1 colleges joined Achieving the Dream, but one college subsequently dropped out. 
’Brock et al. (2007). 

*Brock et al. (2007). 
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individual teaching methods or work ethic.® However, by the fourth year of the initiative, all but 
three colleges reported an increased respect for data analysis at their institutions, with many 
attributing this improvement to their work in Achieving the Dream. Through their colleges’ 
involvement with Achieving the Dream, many faculty and staff learned that data from their IR 
departments could provide useful information on their students’ performance. As one faculty 
member acknowledged, “I know who [our IR director] is now, and I know what her office does. 
I know that she’s accessible; I know that if I want data, all I have to do is. . .explain what kind of 
data I’m looking for.” 

Development of Data Capacity 

By spring 2009, most colleges had made noticeable progress in developing their data 
capacity. All but eight of the Round 1 colleges (69 percent) had improved their data capacities 
or reported an improved ability to meet their basic data needs. As shown in Figure 3.1, half of 
the colleges reported increasing IR staffing, either by hiring new staff or by reassigning current 
staff to work on data collection and analysis efforts. That increase in staffing allowed those 
schools to diversify their data collection and networking with other faculty and staff. For 
example, a number of colleges were able to assign individual staff members to specialized tasks, 
such as developing data codebooks, managing data warehouses, conducting focus groups, or 
running advanced statistical analyses to help predict student trends. Increased staffing has also 
freed up IR directors’ time at some schools, allowing them to dedicate more time to conducting 
higher-level research, making data-related presentations on campus, or spearheading college- 
wide decision-making committees and taskforces. 

A number of colleges also focused on improving their IR staff s level of expertise. Just 
over one-fourth of the Round 1 colleges provided further training for their IR staff so that they 
could undertake more sophisticated data tasks (Figure 3.1). For instance, when expanding its IR 
department from two to four staff members, Broward College (formerly Broward Community 
College) provided statistical training in SAS (a widely used data management software pro- 
gram) to all of these individuals. Several other colleges noted that expanding the number of staff 
with training in data and research helped them avoid overburdening other IR staff. Most often, 
this training focused on helping IR staff manage large databases more efficiently, quality-check 
the data to ensure its integrity, or consolidate data from various institutional sources, such as 
from academic departments and student service centers. 



®Brock et al. (2007). 
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Achieving the Dream: Community Colleges Count 



Figure 3.1 

Progress of Achieving the Dream Round 1 Colleges in Developing Data and Research 

Capacity, Spring 2006 to Spring 2009 




Better met basic Increased IR staff Improved IT Increased IR Used scorecards or 
data needs infrastructure training dashboards 



Type of Institutional Research or Data Improvements 



SOURCE; Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facihtators. 

NOTES; “IR” stands for institutional research; “IT” stands for information technology. 

“Data scorecards and dashboards are tools used by colleges to provide streamhned, user-friendly informa- 
tion about key performance outcomes. 

Another way in which colleges increased their capacity to use data was by improving 
their IT infrastructures. Just under half of the Round 1 colleges reported investing resources in 
their IT infrastructures in order to facilitate their data collection, analysis, and dissemination 
efforts (Figure 3.1). For some colleges, this effort involved the development of large data 
warehouses or databases to store various forms of institutional and student data submitted from 
academic departments, student service centers, or other campus offices. Other colleges focused 
on upgrading statistical software and Web portals in order to allow the campus community to 
access “real-time” data from their own desktops or more easily submit data requests. Some 
colleges also created “scorecards” and “data dashboards” to involve more faculty and staff by 
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providing streamlined, up-to-date information about key outcomes of interest (Figure 3.1). As 
one IR director explained, “Instead of having to create a PowerPoint each time, [data are] all 
collected on the scorecard,” which provided more accessible data. Finally, a few colleges 
created cross-functional, cross-departmental data systems to leverage IT resources for data 
management and reporting while sharing the responsibilities of research, assessment, and data- 
driven programming across areas. For example. South Texas College and Tallahassee Commu- 
nity College both reorganized their IR departments to make data collection and analysis more 
accessible and functional for faculty and staff. 

Though fewer in number, some Round 1 colleges continued to have substantial prob- 
lems with data analysis after five years in the initiative. In total, just over one-fourth of the 
Round 1 colleges still had very weak IR capacity. Some simply did not have the proper person- 
nel or resources in place to develop an IR department. These schools tended to employ only one 
IR staff member to handle all aspects of data and research, often without any clerical or techni- 
cal support. Other colleges could not upgrade their archaic data systems, thus limiting their data 
collection and analysis. One college — with a lack of both personnel and technology — was 
only able to provide raw data for administrators who requested data reports. Even with the 
assistance and training provided by Achieving the Dream, the difficulties of hiring, retaining, 
and training the proper personnel at some of the smaller, underfunded colleges proved to be an 
insurmountable impediment to further developing a culture of evidence. 

Colleges’ Data Collection and Analysis Activities 

While few Round 1 colleges employed sophisticated data analyses early on in the initia- 
tive,'” a number of them had developed the capability to undertake this work after five years in 
Achieving the Dream. By spring 2009, more than three-fourths of the Round 1 colleges had 
collected longitudinal data, and over half had attempted some type of analysis on these data (see 
Figure 3.2). Furthermore, approximately half of the colleges were making use of longitudinal 
data as part of their collegewide discussions around practice and policy development. Many of 
these schools had the capability to compare and track different cohorts of students from se- 
mester to semester, and they used this information to assess whether a particular intervention 
made any difference in student outcomes such as performance and persistence. Other colleges 
utilized their longitudinal data to determine student behavioral trends across time, such as any 
declines in course loads, grade point average, or retention rates across various student groups. 

Additionally, most colleges (85 percent) collected data disaggregated by race, and over 
half (69 percent) reported using their disaggregated data to identify where achievement gaps 
existed (Figure 3.2). A few colleges noted that these disaggregated data helped them see 

'"Brock et al. (2007). 
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Achieving the Dream: Community Colleges Count 
Figure 3.2 

Development of Diversified Data Activities at Achieving the Dream Round 1 Colleges, 

Spring 2009 




Type of Data Activity 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 



noticeable demographic shifts in their student populations over recent years, which led them to 
reconsider their existing programs and services for these student constituencies. Other colleges, 
however, did not seem to find much use for data disaggregated by race or income, because their 
student populations were more homogenous. Instead, some decided to disaggregate data by 
other categories of interest, such as level of college preparedness (for example, developmental 
education status) and student status (for example, first-year students). 
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Finally, all of the colleges were also conducting some form of qualitative data analysis 
by the fifth year of the initiative. Most of the colleges used surveys or focus groups to gather the 
perspectives of students, faculty, staff, or external stakeholders. (See Chapter 4 for more 
information.) Some schools undertook more intensive qualitative efforts, such as Santa Fe 
Community College’s qualitative study on Hispanic men and Danville Community College’s 
efforts to conduct focus groups with its African American male students to better understand 
what they wanted and needed in college. 

Colleges’ Use of Data to Make Decisions About Programs and Policies 

After five years in Achieving the Dream, a majority of the colleges still had difficulty 
using their evaluations of their programs to advance more effective, evidenced-based strategies. 
Just over 40 percent of the Round 1 colleges reported conducting regular evaluations of their 
Achieving the Dream strategies and monitoring how those interventions affected student 
outcomes (see Figure 3.3). While most of the other colleges attempted to evaluate some of their 
strategies, their efforts were generally hampered by difficulties with their data systems or 
limited IR staff time. A number of colleges described the challenge of balancing their responsi- 
bilities to meet the increased demand for data reports with the need to make time for quality 
assessments of new programs. Other stumbling blocks included delays or changes to strategy 
implementation plans, which limited the type of evaluations that could be undertaken. 

Despite this, a few of the colleges with stronger data systems did establish data-driven 
guidelines and procedures for developing their program strategies. For example, some of these 
colleges established standard procedures for faculty and staff to generate a program idea, 
explain its rationale, provide evidence to support that rationale (usually based on institutional 
data), and describe how data could be used to determine the effectiveness of the proposed 
program. In some instances, this required interested faculty and staff to work with IR staff as 
part of a task force to track the success of a program and make recommendations for necessary 
changes. In other cases, college staff worked with local community or K-12 partners, sharing 
data on students’ performance across institutions. 

Finally, some colleges used strategy evaluation data to make decisions about program 
scale-up. Often using simple comparisons between students who participated in a strategy and 
those who did not, these colleges generally sought to expand strategies that were more success- 
ful and revise those that were less promising. For instance, Houston Community College 
pointed to pilot data indicating that students in its Guided Studies “student success” course 
showed noticeable increases in retention and engagement over peers who did not participate in 
the course during their first semester. These promising pilot findings led administrators to scale 
up the course districtwide to all students who were in college for the first time. On the flip side, 
another strategy that Houston attempted — which focused on providing financial aid 
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Achieving the Dream: Community Colleges Count 



Figure 3.3 

Evaluation of Strategies for Improving Student Success at 
Achieving the Dream Round 1 Colleges, Spring 2009 




Evaluated strategies Attempted to evaluate Did not evaluate strategies 

strategies 



Success with Evaluating Strategies 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 



information to the local, non-college-going community — was discontinued as a result of less 
promising results. Box 3.2 provides additional examples of strategy evaluations in Achieving 
the Dream colleges. 



Usefulness of Achieving the Dream’s Inputs 

Many of the Round 1 colleges noted that their participation in Achieving the Dream had 
improved their knowledge of, respect for, and use of student outcomes data. Nearly all of these 
schools said that Achieving the Dream had improved their respect for data analysis and its 
utility for understanding more about their students’ challenges. Additionally, the initiative had 
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Box 3.2 



Using Data to Build Interventions: Case Studies of Three Colleges 

Colleges’ evaluation efforts are a key component of the data-informed decision-making 
process prescribed by Achieving the Dream. The initiative partners recommend that col- 
leges use both quantitative and qualitative data to create evaluation plans for their strate- 
gies. These plans are designed to assess the extent to which a particular intervention has 
improved student success and to inform decision-making around current and future strat- 
egies. However, numerous colleges struggled with the development and enactment of 
evaluation plans. The examples below suggest several ways in which colleges have used 
strategy evaluations to further their work. 

First, several colleges have used student outcomes data to identify problem areas and tar- 
get groups, and then conducted focus groups to determine which policies and programs 
should be introduced. For example, in analyzing its developmental education students, El 
Paso Community College discovered that 98 percent of its incoming students needed 
remedial math education. In response to these data, the college began to hold public meet- 
ings with representatives from local school districts and the University of Texas at El Paso. 
These public meetings led to the launch of a College Readiness Consortium, which estab- 
lished a six-step assessment protocol that included comprehensive orientation, preparation, 
and testing of high school students, followed by post-test interventions, retesting, and 
“summer bridge” programs (programs for high school graduates to prepare them for col- 
lege) to help students place at college level on the Accuplacer college placement test (a 
computerized placement tool designed to provide placement and advising information for 
students entering college). Evaluation results indicated an increase in the number of stu- 
dents placing into college-level math, reading, and writing.^* 

Other colleges used data derived from intervention pilots to determine the potential effec- 
tiveness of these strategies and which ones should be implemented at full scale. Eor exam- 
ple, Tidewater Community College piloted its EOCUS orientation program to determine 
whether it was successful at increasing the number of students enrolling in the college’s 
student success courses during their first semester. The EOCUS orientation was piloted 
with students who were the first in their family to attend college, targeting those who were 
most in need of college-level skills. Tidewater’s evaluation of this pilot revealed promising 
improvements in the persistence of students who participated in the EOCUS orientation 
program. As a result, the college chose to fully scale up the orientation program to reach 
larger numbers of entering students. 

To address ongoing evaluation needs, several colleges have created permanent committees 
to monitor and direct their student success efforts. Eor example, Hillsborough Communi- 
ty College established the Student Success, Retention, and Placement Committee, a facul- 
ty-led body that uses data and research to examine issues related to student success. As 
part of its work, this committee noted that students who enrolled late and missed the first 
day of classes tended to have lower success rates than those who began the semester on 
time. As a result, the college chose to revise its late enrollment procedures. 



“Kerrigan and Slater (2010). 
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an important influence on a number of colleges’ research capacity, with over one-third of these 
institutions noting that they used their Achieving the Dream grants to increase their IR staffing. 
Others found that Achieving the Dream had improved the collaboration between their IR and IT 
staff and encouraged them to collect more diversified types of information about their student 
populations. For instance, nearly 40 percent of these schools attributed their increased use of 
qualitative research to their participation in Achieving the Dream. 

While Achieving the Dream appeared to have a positive influence on colleges’ overall 
data capacity, the initiative’s specific inputs were more mixed. For instance, a majority of the 
Round 1 colleges (16 colleges) found their data facilitators to be helpful in guiding their 
research practices; however, a substantial number (10 colleges) had complications with these 
individuals that made their journeys through the initiative more challenging (see Figure 3.4). 
The data facilitators at colleges with successful experiences were seen as making important 
contributions to colleges’ understanding of new systems or tools to improve their data analysis. 
Data facilitators were seen as particularly helpful in sharing new tools for communicating about 
data, such as institutional scorecards or dashboards, or in helping colleges to develop more 
sophisticated data analyses. A couple of colleges also noted that their data facilitators helped 
them to develop analyses for their accreditation reports. Moreover, some colleges reported 
learning from their data facilitators about how to effectively use data from focus groups and 
surveys to complement information in their quantitative analyses of student outcomes. 

Virtually all of the schools that had difficulty with their data facilitators noted that they 
were not a “good fit” for their data needs. Several colleges that began the initiative with fairly 
weak data capacities noted that their data facilitators assumed that they knew more than they 
did, and consequently did not provide enough concrete assistance in furthering their data 
knowledge. Other colleges noted that their data facilitators were overly prescriptive or critical or 
insufficiently available, all of which made their data work more challenging. While five of these 
colleges requested and received new data facilitators who were more helpful than those who 
were assigned originally, others continued to muddle through with their original data facilita- 
tors. Unfortunately, nearly all of the colleges that stmggled with their data facilitators also had 
weaker IR departments at the outset of Achieving the Dream, which may have further exacer- 
bated their challenges. While a majority of these schools were able to overcome their difficulties 
and build strong IR systems, a few continued to struggle with building their data capacities. 

Achieving the Dream’s database, which was intended in part to help support colleges’ 
longitudinal data analysis, received more critical reviews (Figure 3.4). Most colleges (88 
percent) reported that they did not find data from the Achieving the Dream database to be 
helpful, nor did they view the process of submitting these data as “useful” for developing their 
knowledge (85 percent). First, colleges noted a number of challenges with the “long, arduous” 
process for submitting these data, which several characterized as “frustrating,” “demeaning,” 
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Achieving the Dream: Community Colleges Count 
Figure 3.4 

Contributions of Achieving the Dream Inputs to Use of Data Among Round 1 Colleges, 

Spring 2009 



■ Useful 




Achieving the Dream Inputs 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 



and “burdensome.” Several IR staff emphasized that the differences between the templates for 
the Achieving the Dream database and their own data systems required them to expend exten- 
sive staff time and cost to align the two systems. They also explained that annual changes made 
to these templates created an extra burden, as they had to continually revise their programming 
and submissions. 

Additionally, most colleges found that the database itself proved less useful for tracking 
their institutional progress than their own internal data. For instance, a number of schools noted 
that information from the Achieving the Dream database was not as detailed or relevant as their 
own institutional data in helping them understand their problem areas. Additionally, because 
Achieving the Dream only collected data on new students who entered in the fall semester and 
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did not specify students’ participation in intervention strategies, many colleges felt that the data 
presented an inaccurate picture of their achievement and progress, particularly when compared 
with their own internal tracking. Such challenges made it difficult for most colleges to utilize 
these data to support their reform efforts. 



Summary 

As part of Achieving the Dream, the Round 1 colleges were expected to build a culture 
of evidence by making use of student success and institutional progress data. For some of these 
colleges, this effort entailed dedicating more staff and resources to their IR departments. For 
others, it meant enhancing their already well-established IR departments by increasing the skill 
sets of their staff to undertake new methods for collecting and analyzing data, or by updating IT 
functions and capabilities to better facilitate data activities. Many colleges also moved beyond 
basic data collection and analysis to focus on using data to learn about achievement gaps and 
the success of their programs and strategies. This requirement meant that colleges often had to 
track student performance and persistence outcomes from semester to semester in order to 
pinpoint when they needed to provide more support to prevent students from dropping out or 
failing. 



While the Round 1 colleges achieved noticeable success in increasing their data collec- 
tion and analysis activities, a number of colleges continued to be hampered by ineffective data 
systems or a lack of IR personnel to undertake the data analyses required of them. While some 
strong colleges got stronger, a number of colleges with weaker data capacities failed to institute 
large-scale systems for acquiring and managing data. Given this situation. Achieving the Dream 
might wish to consider what additional supports it can provide to colleges in these areas. For 
instance. Achieving the Dream might wish to assign its strongest data facilitators to colleges 
with weaker data capacity and provide more structured training to help those coaches deal with 
their schools’ unique challenges. 

Additionally, Achieving the Dream might consider providing specialized help to those 
colleges that are unable to hire sufficient staff to manage their data needs. This approach may be 
particularly valuable for colleges that are located in mral areas, as these schools, in particular, 
noted challenges with hiring skilled IR staff. Finally, Achieving the Dream might wish to offer 
more advice to financially strapped schools on how to support the often high costs associated 
with hiring new staff or obtaining new data systems. 
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Chapter 4 

Broad Engagement 



While leadership commitment and data analysis are key steps in community colleges’ 
efforts to increase student success, broadly engaging faculty and staff in an institutionwide 
improvement process is critical for changing students’ achievement levels. Faculty, who are 
responsible for teaching the content knowledge that students need to master their college 
courses, and student services staff, who provide the counseling and advising that helps students 
navigate their college careers, both play key roles in fostering students’ success in college. 
Indeed, many education reform movements have focused centrally upon improving these 
individuals’ practice as the most concrete way of increasing students’ achievement.' 

In addition to full-time faculty and staff, other stakeholders can play an important role 
in students’ achievement. At community colleges, adjunct, or part-time,^ faculty play a particu- 
larly important role in students’ success, as these personnel generally teach the majority of the 
developmental and entry-level classes that students are most likely to take when they enter 
college.^ Additionally, community colleges often have strong ties with their surrounding 
neighborhoods and districts, and changes in their local environment can have an important 
effect on these colleges’ work."' For instance, community colleges can play an important role in 
training workers when new industries move into the area or in providing additional education to 
laid-off workers in a depressed economy. Finally, the students who attend community colleges, 
and their experiences and challenges, can also play a key role in shaping colleges’ efforts to 
improve their skills. 

Achieving the Dream has long noted the critical role that these internal and external 
stakeholders play in colleges’ institutional improvement process. From early in its development, 
the initiative has pushed for faculty and staff to play a central part in improving student 
achievement, noting that, 

[A] college cannot work intensively to raise student outcomes without buy-in 
from faculty and staff. Institutional change means changing the behavior of 



'Eknore and Burney (1996); Datnow and Castellano (2001); Meier (2002); Mumane and Levy (1996). 
^The terms “adjunct” and “part-time” are used interchangeably in this report to describe individuals who 
are not employed as full-time faculty at the colleges. 

^Wallin (2005). 

"'Chen (2009); McClure (2010). 
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people across the institution, and their support will come most readily when 
they share responsibility for diagnosing the problems and crafting solutions.^ 

The initiative has also focused on bringing the perspectives of students and outside 
community members into this change process as well, arguing that colleges must “understand 
the obstacles their students face in order to design appropriate solutions” for increasing their 
success.® As such, the engagement of faculty, staff, students, and external stakeholders is 
highlighted in the third step of Achieving the Dream’s model for institutional improvement (see 
Figure 1.2 in Chapter 1). Additionally, in 2009, the initiative delineated a number of important 
practices associated with “broad engagement” that colleges should seek to emulate as they 
develop and implement their reforms (see Box 4.1). Key among those practices are faculty and 
staff members’ regular use of student outcomes data and outside research on effective practices; 
intentional collaboration between faculty and staff in developing new strategies for improving 
student success; and the inclusion of part-time faculty, students, and external stakeholders in the 
colleges’ change process. 

While a focus on broad engagement has long been a part of Achieving the Dream, the 
initiative has only recently codified these practices and supports into a framework comprising 
four principles and thirty-four practices.’ As such, the Round 1 colleges have been aware of 
Achieving the Dream’s theory of broad engagement during their time in the initiative but have 
had more limited exposure to the specified practices outlined in the initiative’s four-principle 
model. Thus, while this chapter discusses colleges’ development of engagement practices, it 
offers a look at the initiative’s early influence, before all of its practices and supports had fully 
solidified. 

The sections that follow outline the progress of the 26 Round 1 colleges in engaging 
both internal and external stakeholders in their efforts to improve students’ achievement. In 
general, most of the Round 1 colleges tended to involve their faculty and staff in the develop- 
ment and implementation of strategies aimed at improving student success. Many colleges also 
engaged faculty and staff in committees tasked with leading the colleges’ reform process; 
however, their involvement in these leadership committees tended to be limited. Most faculty 
and staff at the Round 1 colleges also had at least some exposure to student outcomes data, with 
many using institutional-level data, such as enrollment or graduation rates, and classroom 
measures, such as grades and course evaluations, at least once per year. However, fewer faculty 
and staff used student outcomes data regularly in their everyday practice or analyzed students’ 
achievement disaggregated by race, ethnicity, and income. 



®MDC, Inc. (2006), p. 9. 
®MDC, Inc. (2006), p. 9. 
’Achieving the Dream (2009). 
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Box 4.1 



Principles and Practices for Increasing 
Student Success Through Institutional Improvement: 
Practices Associated with Principle 3, Broad Engagement 

Faculty and Staff 

• Faculty meets regularly to examine course and program outcomes and develop strate- 
gies for addressing achievement gaps and improving student success. 

• Faculty routinely assesses academic programs and teaching strategies from the per- 
spective of current research on effective practice. 

• Part-time or adjunct faculty are actively engaged in institutional efforts to improve 
student success. 

• Student services staff routinely assess student success strategies from the perspective 
of current research on effective practice. 

• Efforts to improve student success involve alignment and extensive collahoration 
between academic/instmctional units and student services. 

Students and External Stakeholders 

• College secures active student partieipation in efforts to improve student outcomes. 

• College seeures input from external stakeholders to identify eauses of aehievement 
gaps and inform the development of strategies for improving student success. 



SOURCE: Field Guide for Improving Student Success (2009), from the Achieving the Dream: 
Community Colleges Count initiative. Available online at 

www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf 



The involvement of other stakeholders in the Round 1 colleges’ institutional improve- 
ment process tended to be mixed. Many Round 1 colleges involved students and external 
stakeholders in multiple aspects of their work. However, the involvement of adjunct faculty was 
a particular weakness for these schools, with only a handful of schools reaching out to those 
professionals. 



Levers for Change in Achieving the Dream 

Achieving the Dream has provided several supports to help colleges foster the engage- 
ment of their faculty, staff, students, and community members in their student success work. 
First, Achieving the Dream encourages colleges to involve their faculty, staff, and students on 
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committees and leadership teams, including the members of both the core and data teams who 
are involved in implementing the colleges’ Achieving the Dream plans. The core team, which is 
charged with leading the college’s institutional change agenda, is expected to include the 
president, chief academic officer, chief student services officer, institutional research (IR) 
director, and faculty leaders. The data team, which is charged with conducting analyses of the 
college’s performance and student outcomes, is expected to include the IR director, faculty, and 
staff.* Additionally, the coaches and data facilitators, whom the initiative employs as consultants 
for the colleges, are expected to assist colleges in developing plans to actively involve those 
players in the colleges’ reform process. As described earlier. Achieving the Dream also pro- 
vides several different opportunities for participating colleges to convene so they can share their 
ideas and improvement strategies, such as a beginning Kick-off Institute and an annual Strategy 
Institute, and encourages colleges to send faculty and staff to those events.® 

The initiative has also provided some formal structures to help colleges engage com- 
munity members, primarily through the assistance of Public Agenda, a national organization 
with expertise in community engagement. Throughout the course of the initiative, six Round 1 
colleges collaborated with Public Agenda to gather the perspectives of both internal and 
external stakeholders and integrate their views into their improvement process. Additionally, in 
2007, Public Agenda developed a tool to assist colleges in holding conversations with external 
stakeholders,'® which includes guidance on techniques such as focus groups, stakeholder 
dialogues, and community conversations." 

While Achieving the Dream has provided some formal and informal processes for in- 
volving internal and external stakeholders in colleges’ institutional improvement efforts, the 
initiative generally did not provide formal coaching for faculty, staff, or other stakeholders. The 
initiative also did not tend to prescribe what colleges should do in the areas of professional 
development for faculty, data analysis training, or outreach to adjuncts. Instead, higher-level 
college administrators, with guidance from their Achieving the Dream coaches, were primarily 
responsible for communicating the college’s institutional improvement plan to faculty, staff, 
students, and external stakeholders and devising plans to involve them in those efforts. 



*MDC, Inc. (2008). 

®MDC, Inc. (2008). 

'“Freidman (2007). 

"other tools and guides were created in mid-2009 to assist community colleges with engaging internal 
and external stakeholders; however, those guides were produced after the Round 1 colleges’ demonstration 
phase and thus were not available during the years covered in this report. 
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Engagement of Faculty and Staff 

Like the higher-level administrators at their colleges, a large majority of the faculty and 
staff leaders at the Round 1 Achieving the Dream colleges believed strongly that their students 
could succeed, as described in Chapter 2. However, faculty and staff tended to be more in- 
volved in certain aspects of their colleges’ improvement agendas than others. 

Involvement of Faculty and Staff: Strategy Development 

Overall, faculty and staff tended to be more heavily involved in developing and imple- 
menting their colleges’ interventions and strategies aimed at improving students’ achievement 
than in any other part of their colleges’ Achieving the Dream work. The overwhelming majority 
of colleges (over 80 percent) involved at least a moderate number of full-time faculty and staff 
in developing their intervention strategies (see Figure 4.1). Although some colleges involved 
faculty in a multitude of interventions, most colleges tended to have a handful of faculty 
engaged in implementing a select group of strategies. Student services staff tended to be even 
more heavily involved in colleges’ efforts than faculty, so much so that some staff noted feeling 
“overburdened” by their colleges’ Achieving the Dream efforts. Most colleges with high or 
moderate staff involvement tended to engage student services staff in two or more interventions, 
which generally included a mixture of intensive advising, student success courses, revised or 
new orientations, or new registration procedures. 

In general, smaller colleges tended to be better at involving their faculty in strategy de- 
velopment than were larger colleges. For instance, four of the colleges with the heaviest faculty 
involvement had fewer than 2,500 students. In contrast, staff involvement in strategy develop- 
ment tended to be unrelated to college size, with many mid-sized and large colleges having staff 
members who were heavily involved in strategy implementation.^^ While only two larger 
colleges were able to effectively engage large numbers of faculty, certain attributes of colleges’ 
strategy development helped facilitate their involvement (see Box 4.2). Very few colleges 
reported low levels of faculty and staff involvement in strategy development. Those that did 
tended to center on faculty or staff resistance to Achieving the Dream, a focus on instructional 
strategies, or the limited scale or scope of their college’s strategies as reasons for staff’s and 
faculty’s limited involvement. 



*^The colleges tend to employ fewer student services staff than faculty, which may explain why services 
staff had higher levels of engagement in the colleges’ reform efforts. 
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Achieving the Dream: Community Colleges Count 
Figure 4.1 

Faculty and Staff Involvement in Student Success Reforms at the 
Achieving the Dream Round 1 Colleges, Spring 2009 
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SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 
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Box 4.2 



Promising Practices for Promoting Facuity and Staff Engagement 

While faculty engagement is a core component of Achieving the Dream, some colleges 
had challenges with engaging large numbers of faculty and staff in their reform efforts. 
However, several colleges were able to overcome these difficulties by creating more sys- 
tematic ways to facilitate faculty and staff involvement in the development of their strate- 
gies, committee structures, and data analyses. 

While most colleges involved faculty and staff in implementing their strategies, larger col- 
leges tended to have greater difficulties involving substantial proportions of their faculty in 
these efforts. One of the most successful ways that a few colleges engaged greater numbers 
of faculty in strategy development was by implementing large-scale interventions that 
touched a number of different academic areas. For instance, Houston Community College 
developed a success course for its first-time students, which involved a sizeable group of 
faculty and staff teaching student success courses. Because it was such a large-scale strate- 
gy, Houston’s success courses required more faculty and staff involvement than many other 
smaller-scale strategies, which touched only certain subsets of students. 

In order to increase faculty and staff involvement in decision-making, a number of the 
Round 1 colleges developed integrated committee systems focused on student success. For 
example, Hillsborough Community College created the Student Success, Retention, and 
Placement Committee, which brought faculty together to analyze data on the college’s 
programs and determine future actions to further student success. This committee would 
then make recommendations about the continuation or revision of the college’s interven- 
tions. Another college, Guilford Technical Community College, created a similar com- 
mittee, which was also tasked with monitoring and revising the college’s student success 
efforts using evaluation data. Guilford rotated membership on this committee every two 
years in order to facilitate greater faculty and staff involvement. 

Other types of strategies for involving faculty and staff in data analysis emerged from the 
colleges that had high levels of engagement. One of the most common practices was the 
creation of scorecards or data dashboards, which summarized students’ achievement in 
particular departments or subject areas. Rather than focus solely on institutional-level 
measures, colleges that developed these tools were able to clearly present student success 
data, such as course pass rates and persistence, on a departmental level. This more indivi- 
dualized approach provided a readily accessible way for faculty and staff to understand 
how their students were doing. Another strategy for involving faculty and staff in data 
analysis was to provide more intensive training in data programming, such as training in 
how to use data software packages or how to access student outcomes data that were 
stored in an institutional data warehouse. For instance, at Valencia Community College, 
training in the SAS software package was undertaken to decrease the data requests that the 
college’s institutional research department received by empowering individual faculty and 
staff to access the data they needed on their students’ achievement. 
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Involvement of Faculty and Staff: Committees 

The involvement of faculty and staff in the leadership of a college’s change agenda is 
also an important aspect of Achieving the Dream’s vision for change. Most Round 1 colleges 
took this advice to heart and were successful at engaging at least some of their faculty and staff 
in committees charged with making critical decisions about the colleges’ reforms (Figure 4.1). 
Colleges with high levels of faculty and staff involvement often restructured their committee 
systems, creating a number of smaller task forces that allowed for greater faculty and staff input 
(see Box 4.2 for examples). These task forces tended to make recommendations on specific 
policies and programs and were integrated with the colleges’ broader leadership teams, which in 
some cases included faculty and staff representatives. 

In general, however, at a plurality of Round 1 colleges, a select group of faculty and 
staff tended to be involved in lower-level committees, such as task forces dedicated to strategy 
development. While a few colleges had faculty and staff on higher-level, policymaking commit- 
tees, most colleges tended to limit these committees to administrators, giving faculty and staff a 
smaller voice in the colleges’ decision-making processes. Another 20 to 30 percent of the 
Round 1 colleges had low faculty and staff involvement, primarily because of faculty and staff 
resistance to Achieving the Dream, difficulties with the colleges’ leadership, or colleges’ greater 
focus on strategy development. 

Involvement of Faculty and Staff: Data Analysis 

One of the most critical aspects of Achieving the Dream’s framework is the wide dis- 
semination of student outcomes data to the larger campus community and the training of faculty 
and staff in the use of data and research in their everyday practice (see Box 4.1). In general, 
most Round 1 colleges tended to heed these suggestions, with the majority of faculty and staff 
at these institutions having at least some exposure to data on student outcomes. Data from the 
implementation visits, as well as a survey of full-time faculty and administrators at 23 of the 26 
Round 1 colleges,'^ revealed that the overwhelming majority of faculty and staff reviewed some 
type of institutional-level data, such as enrollment and graduation rates, and classroom-level 
data, such as grades and course evaluations, at least once per year (see Figure 4.2). Faculty 
tended to review classroom-level data more often than institutional-level data, perhaps because 
this information was more relevant for their classroom practice. However, they also reviewed 
other types of information, such as data related to course placements, retention, and graduation, 
as well as outside research. 



'^Three colleges — Tidewater Community College, Valencia Community College, and Southwestern 
Indian Polytechnic Institute — did not participate in the survey. 



60 




Achieving the Dream: Community Colleges Count 
Figure 4.2 

Frequency with Which Faculty Members Used Student Outcomes Data, 
by Type of Data, 2007 to 2008 
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SOURCE; CCRC calculations using survey data collected from faculty and administrators at the Achieving 
the Dream Round 1 colleges. 

NOTE: Faculty and administrator survey data were collected from September 2007 through February 2008, 
approximately one year before implementation research at the Round 1 colleges. The survey was dissemi- 
nated at 23 of the 26 colleges. 
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Achieving the Dream: Community Colleges Count 
Figure 4.3 

Frequency with Which Faculty Members Used Data Disaggregated hy Race and Income, 

2007 to 2008 




SOURCE: CCRC calculations using survey data collected from faculty and administrators at the Achieving the 
Dream Round 1 colleges. 

NOTE: Faculty and administrator survey data were collected from September 2007 through February 2008, 
approximately one year before implementation research at the Round 1 colleges. The survey was disseminated 
at 23 of the 26 colleges. 

While most faculty and staff had some exposure to student outcomes data, the type of 
data that were analyzed and the regularity of their review were mixed. For instance, most 
faculty and staff had limited exposure to the institutional-level measures that are most important 
to Achieving the Dream, such as graduation and retention rates, with most faculty reviewing 
these data less often than four times a year. Similarly, few faculty and staff regularly reviewed 
data that were disaggregated by race or income, with over 50 percent analyzing these data less 
often than once a year (see Figure 4.3). However, faculty did review other types of data much 
more regularly. For instance, approximately half of the faculty at the Round 1 colleges analyzed 
information on course grades and enrollment rates four or more times per year. Similarly, a 
number of faculty noted using data and research to some extent when thinking about their 
advising practices, curriculum, and teaching. 
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Perhaps some of this variability in data use stemmed from the fact that very few colleg- 
es brought faculty and staff into the data analysis process. Only seven colleges offered some 
form of training on data analysis to faculty and staff or had institutional research departments 
that were regularly involved in teaching faculty and staff how to use data. Several other colleges 
noted that their faculty members were involved in some form of learning outcomes assessment; 
however, these analyses tended to be driven by colleges’ accreditation processes more than by 
their Achieving the Dream efforts. While most colleges delegated data analysis work to their 
institutional research departments, a number of key practices helped facilitate faculty’s and 
staffs understanding and use of student outcomes data, providing a good model for better 
faculty engagement (see Box 4.2). 

Collaboration Between Faculty and Staff 

Achieving the Dream’s model for institutional improvement also emphasizes the need 
for greater collaboration between faculty and student services staff when implementing colleg- 
es’ reforms (see Box 4.1). Overall, Achieving the Dream appears to have positively influenced 
faculty and staff collaboration at the Round 1 colleges, with the majority of the Round 1 
colleges noting an increase in the collaboration between their faculty and staff as a result of their 
participation in the initiative. There were only a few exceptions to this trend, generally resulting 
from the college’s focus on one specific area (either instruction or student services), or from a 
lack of involvement of one of these parties in the Achieving the Dream process. Colleges’ 
higher levels of collaboration were generally attributable to an increased number of faculty and 
staff working together on one or more of the colleges’ strategies or on leadership and planning 
committees. Particular types of strategies, such as student success courses, orientations, learning 
communities, in-class advising, or tutoring and supplemental instmction, tended to foster these 
interactions, primarily because they blended instmction and student support. One college even 
rearranged its departmental stmcture in order to better facilitate this collaboration. 

Adjunct Faculty Involvement 

Achieving the Dream also recommends that colleges seeking to improve student success 
involve their adjunct, or part-time, faculty in the colleges’ reform efforts (Box 4.1). Adjunct 
faculty are typically instmctors who are employed part time by a college to teach one or two 
classes. As such, these instmctors tend to be less connected to the college environment than full- 
time faculty, as they are on campus for only a short period of time each week. However, adjunct 
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faculty play a central role in community college teaching, as they tend to teach the majority of 
developmental and gatekeeper classes that students take when first entering college.'"^ 

Unfortunately, the involvement of part-time faculty was one of the Round 1 colleges’ 
key weaknesses. Only eight of the Achieving the Dream colleges made attempts to include part- 
timers in their improvement process, with most of these colleges offering only minimal, one- 
time professional development sessions or unpaid invitations to participate in the college’s 
projects. Few colleges provided paid time for adjuncts to be involved in professional develop- 
ment or strategy implementation, and over two-thirds of Round 1 colleges made no concerted 
efforts to engage their adjunct faculty. Colleges noted a couple of reasons for the limited 
involvement of adjunct faculty, including a lack of funding to pay adjuncts for additional work 
and difficulty forming partnerships with adjuncts because of their limited time on campus. 

Despite these challenges, a few Round 1 colleges developed some useful tactics for in- 
volving adjunct faculty in their reform work. For instance, Patrick Henry Community College 
developed a uniquely comprehensive model for bringing its adjunct faculty into a schoolwide 
effort to implement collaborative learning in the classroom. As described in Box 4.3, Patrick 
Henry not only supported initial training for these faculty members, but also provided modest 
pay increases and ongoing support to trained faculty to ensure that they used collaborative 
learning techniques in their classrooms. While somewhat less intensive, other colleges made 
similar efforts to involve these instructors through trainings and meetings around particular 
interventions. For instance, Valencia Community College allowed adjunct faculty to teach their 
Learning in Community (LinC) linked courses and opened their trainings to both adjunct and 
full-time faculty. 



Engagement of Students 

While involving faculty and staff is important. Achieving the Dream also recommends 
that colleges engage students in the institutional improvement process, as they may provide a 
unique perspective on the colleges’ programs and services (Box 4.1). This effort was one of the 
most successful that the colleges undertook, as all of the Round 1 colleges brought student 
perspectives into their agenda in some form (see Figure 4.4). The most popular way to gamer 
students’ perspectives was through the dissemination of surveys; all of the Round 1 colleges 
participated in this activity. The most popular survey was the Community College Survey of 
Student Engagement (CCSSE), which gathers information on students’ perspectives on college, 
use of campus services, and challenges to attendance. All of the Round 1 colleges delivered the 



'“'Boylan, Bonham, Jackson, and Saxon (1994); Boylan, Bonham, and Bhss (1994); Gerstein (2009); 
Schuetz (2002). 
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Box 4.3 



Bringing Adjunct Facuity on Board: Promising Modeis 
from Two Round 1 Coiieges 

Most of the Round 1 Achieving the Dream colleges struggled to engage adjunct, or part- 
time, faculty in their student success reform efforts. However, two colleges provided inno- 
vative models for involving these individuals in their work. In scaling up one of its promis- 
ing Achieving the Dream strategies, collahorative learning, Patrick Henry Community 
College provided extensive supports for adjuncts to learn about and implement this teach- 
ing practice. First, Patrick Henry paid adjunct faculty to attend a training session to learn 
about collaborative learning and how to develop it in their classrooms. Additionally, the 
college offered supplementary pay to adjunct faculty who incorporated this pedagogy into 
their classrooms. Finally, it provided ongoing trainings, peer mentoring, and monthly 
meetings to both full-time and adjunct faculty to support their implementation of this new 
model of teaching. While the increased pay for adjuncts was relatively small, Patrick Hen- 
ry represented one of the few colleges that developed a fully integrated professional devel- 
opment system to support both full-time and adjunct faculty in developing a new teaching 
method. 

Paul D. Camp Community College also focused heavily on adjunct faculty through the 
creation of an adjunct academy that brought part-time faculty members together for train- 
ings and other opportunities. The college found this center to be so effective that it ulti- 
mately extended its services to full-time faculty as well. This center provided monthly 
offerings to adjunct faculty to help them learn about the college’s new reforms. For in- 
stance, in one session, adjunct faculty learned about the new Blackboard technology, 
which allowed for online interaction and collaboration among students and instructors, that 
Paul D. Camp was integrating across its course offerings to help support students’ class- 
room learning and management. Another session focused on a review of the college’s 
findings from the LAS SI (Learning and Study Strategies Inventory) survey and a training 
event focused on students’ different learning styles. Paul D. Camp also charged a number 
of its adjunct faculty with leading some of its student success interventions. While training 
for these personnel was less systematic than at Patrick Henry, Paul D. Camp’s work 
helped adjunct faculty feel that there was “no separation” between full-time and part-time 
faculty, and that they were “not made to feel like adjunct faculty.” 



CCSSE to their students at least once during their participation in Achieving the Dream, with 
many colleges disseminating this survey multiple times. Some colleges also used other surveys to 
gather information about students’ thoughts, engagement, and learning styles, including the 
Learning and Study Strategies Inventory (LASSI), Noel-Levitz Student Satisfaction Inventory, 
Survey of Entering Student Engagement (SENSE), and Visual, Aural, Read/write, Ki nesthetic 
(V ARK) inventory of learning styles. Some colleges also took pains to develop their own surveys 



65 



Achieving the Dream: Community Colleges Count 
Figure 4.4 

Number of Achieving the Dream Round 1 Colleges with at Least Some Involvement 
from Students in the Institutional Improvement Process, Spring 2009 
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SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 

for students to gather information about their perspectives on specific offerings at the college, 
such as campus services, courses, or a new strategy that they were piloting. 

Another popular strategy for learning about student perspectives was to conduct focus 
groups. Over 60 percent of the 26 Round 1 colleges discussed their efforts to meet with students 
through small group discussions. Focus groups were sometimes used to get students’ perspec- 
tives on the strategies and interventions that colleges were testing. However, more often, col- 
leges held focus group sessions to understand students’ concerns and challenges more generally. 
A few colleges made mention of how student perspectives had changed their reform efforts. For 
instance, Santa Fe Community College discussed how their students’ challenges led them to 
increase their student support services and create an Office of Student Development Center that 
coordinates student activities and support. 
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Some colleges sought out ways to foster students’ more active participation in planning 
and implementing their college reform agendas. For instance, some colleges involved students 
directly in helping to implement a particular strategy by serving as a peer mentor, tutor, or 
student advisor. Additionally, one-third of the colleges had student representatives on their 
leadership or decision-making committees, such as the board of trustees or the president’s 
council. Therefore, in addition to learning about students’ perspectives, a select group of Round 
1 colleges also allowed students a real voice in the changes that their institutions were pursuing. 

About two-th ir ds of the colleges reported that Achieving the Dream had an important 
effect on student engagement. The increased outreach at most of these schools centered primari- 
ly on gathering information about student perspectives through surveys and focus groups or 
involving students in the implementation of a strategy. However, a few colleges discussed 
important ways that Achieving the Dream had helped them use student perspectives to revise 
their improvement agenda. For instance, at Durham Technical Community College, students 
participating in the “Visions” program (which, as described in Chapter 2, aimed to help low- 
income African American male students overcome obstacles to succeeding in college) led the 
movement to shift the program’s focus on students’ academic performance. Similarly, at Martin 
Community College, students’ requests for more interaction in the classroom led to Martin’s 
focus on collaborative learning and a restmcturing of its accreditation plan to integrate a more 
collaborative teaching approach. 



Engagement of External Stakeholders 

Achieving the Dream also recommends that colleges bring external stakeholders, such as 
K-12 school districts, universities, community members, parents, and businesses, into their student 
success efforts (Box 4.1). Most of the Round 1 colleges succeeded at bringing some of these 
external stakeholders into their work (see Figure 4.5). High school districts and universities were 
common external partners. A majority of the Round 1 colleges were collaborating with these 
institutions through activities such as “early college” high schools, dual enrollment programs, high 
school-to-college “bridge programs,” early college placement assessments, or articulation agree- 
ments with four-year colleges and universities.'^ Perhaps the best example of these partnerships, 
El Paso Community College worked intensively with multiple K-12 school districts to develop a 
multifaceted College Readiness Initiative, which provided early college assessments, advising. 



'^Early college high schools combine high school and college curricula, allowing students to earn a high 
school diploma and an associate’s degree in a compressed period of time. High school-to-college bridge 
programs are generally summer preparation programs for recent high school graduates that focus on increasing 
graduates’ academic skills and making them more college -ready. 
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Achieving the Dream: Community Colleges Count 



Figure 4.5 

Number of Achieving the Dream Round 1 Colleges with at Least Some Involvement 
from External Stakeholders in the Institutional Improvement Process, Spring 2009 




Type of External Stakeholder 



SOURCE; Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 

NOTE: Community members include businesses, nonprofits, community-based organizations, and local 
residents. 

and preparation to high school students. Other colleges, such as Martin Community College, 
University of New Mexico-Gallup, South Texas College, and Tallahassee Community College, 
also had multiple avenues for students to begin preparing for college and earn college credits 
while they were still enrolled in high school. 

Many Round 1 colleges also sought input from local community members, businesses, 
nonprofit organizations, and other interested parties. As described earlier, some schools worked 
with Public Agenda, an Achieving the Dream partner with expertise in community engagement, 
to develop “community conversations,” through which they brought a number of community 
members and parents into the colleges to hear their hopes and desires for their programs and 

*®For more information, see the case study on El Paso Community College in Kerrigan and Slater (2010). 
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their students. Although a number of colleges had been working with community and business 
leaders for several years, Achieving the Dream had at least a moderate influence on nearly half 
of the Round 1 colleges’ external engagement activities. 



Summary 

The Round 1 Achieving the Dream colleges, as a whole, were able to moderately engage 
a number of internal and external stakeholders in their efforts to improve students’ achievement. 
Colleges were most successful at engaging faculty and staff in developing strategies that were 
aimed at improving students’ achievement, and a majority of schools engaged at least some of 
these personnel in committees tasked with leading the colleges’ efforts. Faculty and staff also had 
at least some exposure to data on student outcomes, although the type of data that were reviewed 
and the regularity of their use varied across institutions. The Round 1 colleges were also success- 
ful at gamering the participation of students and external stakeholders in their work, with some 
playing a particularly active role in the colleges’ reform efforts. 

While the Round 1 colleges had many successes engaging faculty and staff, a few limi- 
tations were apparent in their work. Most important, adjunct faculty, who are typically respon- 
sible for teaching many of the developmental and entry-level classes that students take upon 
entering college, tended to be absent from the colleges’ reform efforts. This disconnect means 
that many of the classroom reforms with which colleges experimented may not have reached 
many of the students who could have benefited the most from the colleges’ work. Additionally, 
while faculty and staff had some exposure to data, they did not make regular use of the institu- 
tional-level data on which Achieving the Dream has been most focused. Instead, faculty tended 
to focus on data that were closest to their classroom practice, such as grades and course evalua- 
tions. Finally, faculty and staff tended to have less of a voice in colleges’ leadership and 
policymaking committees, meaning that their influence on colleges’ reform agendas tended to 
be more limited. 

Given these findings. Achieving the Dream might wish to consider additional ways to 
help colleges engage faculty and staff in the leadership and management of colleges’ reform 
efforts. In particular, colleges seem to need more support in engaging adjunct faculty and staff 
in their work. Additionally, colleges may need more guidance on developing concrete ways to 
involve faculty leaders, such as deans and department chairs, in data analysis, as well as in the 
leadership and management of colleges’ reforms. Several colleges have provided strong models 
for involving faculty and staff more broadly in their efforts, which provide good examples for 
the initiative to follow. A more defined approach for bringing faculty and staff into colleges’ 
efforts may help further strengthen their already good work in this area. 
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Chapter 5 

Intervention Strategies 



The preceding chapters have discussed colleges’ efforts to develop a culture of evidence 
by fostering committed leadership, increasing data usage, and engaging a broad spectmm of 
stakeholders. In Step 4 of the Achieving the Dream model, “Implement, Evaluate, and Im- 
prove” (see Figure 1.2 in Chapter 1), colleges build upon these foundations by implementing 
new strategies to help students remain in school, improve their performance, and ultimately 
graduate with certificates or degrees. These intervention strategies play a crucial role in Achiev- 
ing the Dream’s theory of action as the principal pathway through which colleges’ institutional 
efforts affect what students experience on the ground. As such, the initiative’s success at 
improving student outcomes is contingent upon colleges’ ability to implement substantially 
improved programs that reach sufficient numbers of students. 

Overall, the Round 1 colleges implemented a large number of strategies under the aus- 
pices of Achieving the Dream. The majority of those strategies were direct programs and 
services designed to improve students’ college success. While the strategies ranged widely, 
from “light-touch” orientation activities to more intensive curricular reforms, several common 
themes emerged across the 26 colleges: 

• A large majority of strategies were designed to increase academic and social 
support systems for students, while only about one-fourth changed the con- 
tent and delivery of classroom instruction itself. 

• Nearly half of the colleges’ strategies targeted developmental education stu- 
dents, and one-third focused on students in their first year of college. Very 
few strategies targeted students based on their race, ethnicity, or economic 
status, though some colleges indirectly addressed achievement gaps along 
those lines through programs aimed at wider groups of students. 

• Despite the colleges’ notable efforts to scale up their programs and services, 
the majority of strategies reached less than 10 percent of their intended target 
populations. Intensive strategies — defined by longer periods of contact time 
with students — were particularly unlikely to reach large numbers of stu- 
dents, suggesting that colleges faced a trade-off between the intensity and 
scale of their interventions. 

Most colleges had several additional strategies that did not reach students through direct 
contact hours, but rather through (1) changes in college policies; (2) improvements in profes- 
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sional development opportunities for faculty and staff; and (3) collaborative relationships with 
external stakeholders, including local school districts and community members. This chapter 
provides an overview of Achieving the Dream’s theory for strategy development, explores 
major findings from each category of intervention strategies, and concludes with a discussion of 
the colleges’ overall experiences with strategy implementation under the initiative. 



Achieving the Dream Framework for Strategy Implementation 

Step 4 in the Achieving the Dream theory of action represents an ongoing process that 
colleges are expected to follow in order to improve the programs and services that are available 
to their students. During the initial planning year, colleges identify priority areas for improve- 
ment, draw upon available evidence and research to select strategies that show promise of 
improving outcomes in these areas, and design a plan for strategy roll-out. This planning lays 
the foundation for colleges’ work under Step 4 (summarized in Figure 5.1), which includes the 
following actions: 

• Implement selected strategies, beginning with small-scale pilots when possi- 
ble and engaging broad segments of faculty and staff. 

• Evaluate strategies to assess their success at improving student outcomes. 

• Use evaluation results to inform strategy refinement, scaling up interventions 
that yield promising trends in student success, while modifying or eliminat- 
ing those that do not.' 

Implementing strategies is also one of the primary vehicles through which colleges can 
act to reduce achievement gaps for low-income students and students of color, whether by 
explicitly targeting interventions to particular student subgroups or by incorporating concerns 
about equity into the development and implementation of strategies that reach across student 
populations. 

While Achieving the Dream’s theory of action for strategy development is now well de- 
fined, the Round 1 colleges did not always have such definitive direction for strategy implemen- 
tation. The development of interventions that are aimed at increasing students’ success has been 
a central goal of Achieving the Dream from the beginning, and, with its focus on data-driven 
decision-making, the initiative has also emphasized the importance of monitoring and tracking 



'Achieving the Dream (2009). 
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Achieving the Dream: Community Colleges Count 
Figure 5,1 

Step 4 of the Achieving the Dream Institutional Improvement Process: 
Implement, Evaluate, and Improve Strategies 



Implement strategy pilot 




▼ T 





SOURCE: Diagram created based on information from Field Guide for Improving Student Success (2009), p. 
11, from the Achieving the Dream: Community Colleges Count initiative. Available onhne at 
www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 

students’ progress. However, the initiative did not develop a more codified policy for colleges’ 
evaluation practices until 2008,^ nor did it publish a more detailed statement of the five-step 
process for institutional improvement until 2009.^ Therefore, while this chapter discusses the 



^Achieving the Dream (2008). 
^Achieving the Dream (2009). 
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Round 1 colleges’ process for developing strategies, it is important to keep in mind (as was the 
case in the preceding chapters) that these schools were implementing these interventions while the 
initiative itself was still under development. 



Levers for Change in Achieving the Dream 

The Achieving the Dream partners have provided a number of inputs to support colleg- 
es through the rigorous process outlined above. One of the most noteworthy among these is an 
annual conference — known as the Strategy Institute — that brings together teams of presi- 
dents, administrators, faculty, and staff from all cohorts of Achieving the Dream colleges. The 
Strategy Institute fosters communication and collaboration across colleges as they give presen- 
tations on their experiences with strategy development and implementation, network with one 
another around common interests or interventions, and learn from the initiative partners about 
their respective areas of expertise. Several of the initiative partners — including MDRC, the 
Community College Research Center (CCRC), and Jobs for the Future — have also produced 
written materials designed to inform colleges about specific interventions and policies. For 
example. Jobs for the Future has released a number of briefs designed to help colleges navigate 
and influence state policy as they adopt new institutional practices,"* while MDRC published 
two experimental studies of promising strategies during the early years of Round 1 colleges’ 
participation in Achieving the Dream.^ MDRC has subsequently partnered with four of the 
Round 1 colleges to undertake rigorous evaluations of specific Achieving the Dream interven- 
tions (discussed in Chapter 8), though the findings from these studies were not released in time 
to influence the initial selection of strategies at the Round 1 colleges. 

Outside of conferences and publications, the initiative partners have also developed a 
national database for tracking colleges’ intervention strategies. While this database was not 
available when the Round 1 colleges first joined the initiative, it is now accessible to all Achiev- 
ing the Dream colleges, both seasoned and new. Finally, a subset of the Round 1 colleges have 
received support from Public Agenda, another Achieving the Dream partner organization, in 
their efforts to engage stakeholders from their local communities in their student success efforts. 



"*Biswas (2007); Collins (2009). 

^Bloom and Sommo (2005); Brock and Richburg-Hayes (2006). 
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Type and Reach of Strategies 



Scope of This Analysis 

This analysis is not intended to represent the entirety of colleges’ student success ef- 
forts. Many colleges had initiated student success agendas before joining Achieving the Dream, 
while others pursued important activities during the course of their participation that they did 
not associate with the initiative itself. Rather than capturing the full scope of colleges’ student 
success efforts, this chapter focuses specifically on those strategies that were implemented 
under the auspices of Achieving the Dream. Disentangling colleges’ Achieving the Dream 
strategies from efforts that were undertaken independently of the initiative was a challenge, 
particularly when colleges made conscious efforts to align their Achieving the Dream activities 
with work under other grants and initiatives. As such, researchers from MDRC and CCRC have 
relied heavily on colleges’ own reports about which activities were shaped or significantly 
influenced by their participation in the initiative. 

Direct Programs and Services 

One of the preeminent findings from this analysis is the sheer quantity of new programs 
and services that colleges implemented under Achieving the Dream. In total, the 26 Round 1 
colleges undertook nearly 200 direct interventions in the realms of student services and instruc- 
tion. Though colleges ranged significantly in the number of direct strategies they implemented 
(from three to twelve), the average college implemented seven different interventions. 

Types of Direct Strategies 

Despite the variety of their strategies, the Round 1 colleges also showed striking com- 
monalities in the types of programs and services they pursued. Figure 5.2 illustrates the relative 
popularity of various types of interventions, each of which has been classified as a student 
support service (in light gray), an instructional support (in dark gray), or a change in classroom 
instruction (in white). Overall, it is apparent that a large majority of direct strategies focused on 
the former two categories, providing additional support for students inside or beyond the 
classroom rather than changing the content and delivery of classroom instmction. Examples of 
popular support strategies include Southwestern Indian Polytechnic Institute’s one-stop student 
support center, which offers tutoring and a number of other services to students, and Mountain 
Empire’s Ereshman Orientation Experience, which sought to introduce entering students to the 
college before enrollment. More specifically, programs that were related to tutoring or supple- 
mental instmction emerged as the most popular type of intervention, including 42 strategies 
across 21 colleges. Three types of student support services were also implemented at a majority 
of the Round 1 colleges: advising interventions, student success courses (which provide 
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Achieving the Dream; Community Colleges Count 
Figure 5.2 

Popularity of Direct Program and Service Strategies: 
Percentage of Round 1 Colleges That Implemented Each Type of Strategy 
Under Achieving the Dream 

Bridge programs 




financial aid 



SOURCES: Information collected from faculty, staff, and administrators during two rounds of implementation 
visits across all 26 Round 1 colleges, cross-referenced against intervention reports submitted by colleges to the 
Achieving the Dream partners. 

NOTES: All 187 direct intervention strategies from the Round 1 colleges are classified according to 11 
mutually exclusive strategy types. These types each correspond to one of three broad categories of strategies: 
student support services, instructional supports, or changes in classroom instruction. 



students with fundamental tools for doing well in college, such as time management skills and 
information about campus support services), and new student orientation programs. 

In contrast with the prevalence of student support strategies, only about one-fourth of 
direct strategies aimed to change classroom instmction itself. The relative scarcity of classroom- 
based reforms is consistent with the challenge of engaging broad segments of faculty in colleg- 
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es’ Achieving the Dream work (discussed in Chapter 4).'^ A number of factors might help to 
explain this general trend. At the institutional level, shared governance models at the communi- 
ty colleges often require time-intensive approval procedures before interventions that require 
faculty buy-in can be implemented, thus making it difficult for administrators to make timely 
changes on the instructional side. Initiative-level factors might also have contributed to the 
distribution of strategies presented in Figure 5.2. For instance, the Achieving the Dream theory 
of action takes an expansive view of institutional reform, focusing heavily on the role of senior 
leadership and defining success in terms of broad performance measures such as student 
progression and completion. While these measures are undoubtedly of interest to most faculty, 
they do not provide concrete measures of student learning in the classroom, such as grades and 
student learning outcomes.’ The latter is an expanding area of work for faculty nationwide, as 
well as a central focus of reviews by all of the regional accreditation agencies.* In fact, faculty at 
several of the Achieving the Dream colleges have become involved in setting and assessing 
student learning outcomes, but those activities have taken place largely outside the scope of the 
initiative. It is possible that faculty would engage in Achieving the Dream more quickly if the 
initiative were to incorporate classroom-level measures, such as student learning outcomes, 
along with institutional performance indicators. 

In spite of these challenges, a handful of strategies were designed to influence instmc- 
tional practices — especially with the goal of fostering active and collaborative learning 
techniques — while a larger set focused on changes in course curricula. In particular, learning 
communities stood out as the only type of curricular reform to gain broad popularity across the 
Round 1 colleges, with 15 of the 26 colleges trying this strategy in some form.'* This widespread 
emergence likely reflects the increased spotlight that was placed on learning communities 
during the early years of Achieving the Dream. For example, MDRC released its report on 
promising findings from learning communities during the summer of 2005,'° just as the Round 



°Only six of the Round 1 colleges achieved a high level of faculty involvement in strategy development, 
while twice as many colleges had a high level of involvement from student services staff. 

’in a survey of student data usage across 41 Round 1 and Round 3 colleges, 90 percent of faculty reported 
reviewing grades at least once per year, and 85 percent considered grades to be a useful source of information. 
By contrast, the percentages of faculty who used information about institutional retention and graduation rates 
hovered at 72 and 65 percent, respectively, and faculty tended to view this information as less useful in their 
work (Jenkins and Kerrigan, 2009). 

^Manning (2009). 

^Learning communities reflect an intervention that is designed to strengthen classroom engagement by 
fostering peer relationships and building thematic connections across subject areas. A small group of students 
enrolls in two or more classes together as a “community,” while the instructors of these “hnked” courses work 
together to build integrated syllabi and develop mutually reinforcing themes (Visher, Wathington, Richburg- 
Hayes, and Schneider, 2008). Figure 5.2 presents learning communities separately from other curricular reform 
interventions in order to highlight their relative popularity. 

'°Bloom and Sommo (2005). 
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1 colleges were completing their initial planning year and preparing for their first year of 
implementation under the initiative. Learning communities were also featured at multiple 
Achieving the Dream Strategy Institutes and conferences," as well as at specialized training 
opportunities, including an intensive institute at the Washington Center at Evergreen State 
College. The popularity of this strategy, however, does not imply that it is easy to implement; 
nearly all of the learning communities at the Round 1 colleges have remained limited in scale 
because of the manifold administrative, pedagogical, and financial demands of the program. 

Target Populations for Direct Strategies 

Defining a target population is one of the most important aspects of a college’s strategy 
development efforts, as similar interventions can produce markedly different results, depending 
upon whom they reach. As such, the Achieving the Dream model encourages colleges to 
specify and justify which student group(s) they intend to target with a particular strategy. Figure 
5.3 illustrates that developmental education students were by far the most popular target 
population. Spanning across colleges and strategy types, nearly half of all direct strategies 
focused on developmental education students, with a significant subset targeted specifically to 
developmental students in their first year of college. Furthermore, each of the 26 colleges had at 
least one strategy targeted to developmental education, and three focused all of their direct 
strategies in this area. Some interventions were extended broadly to all students enrolled in any 
developmental course, while others were tailored specifically to students with more remedial 
needs. For example, several colleges, such as Brookhaven Community College, mandated 
student success courses for those students who placed into developmental courses. The domi- 
nance of developmental education as a focus for strategic interventions is reflective of a broader 
national movement to improve services for underprepared students in community colleges. In 
fact, eight of the 26 Round 1 colleges have already begun their participation in the Developmen- 
tal Education Initiative that is funded by the Bill & Melinda Gates Foundation and was 
launched in 2009. 

First-year students were the next most popular target group after developmental-level 
students. Approximately one-third of all direct strategies were focused on first-year students,'^ 
including at least one such strategy at all but three of the 26 colleges. Colleges hoped that by 
reaching students early with targeted interventions — such as new student orientation programs 
and student success courses — they could better prepare students to persist through the chal- 
lenges of their first few semesters. 



"Brock etal. (2007). 

*^This percentage includes 10 percent of direct strategies that simultaneously targeted developmental and 
first-year students. 
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Achieving the Dream: Community Colleges Count 
Figure 5,3 

Popularity of Direct Program and Service Strategies: 
Percentage of Achieving the Dream Strategies Focused on Each Type of 

Target Population 



100 n 



90 - 




Developmental First-year Students in All students Other 

education students gatekeeper 

students courses 



Target Population of Students 

SOURCES: Information collected from faculty, staff, and administrators during two rounds of implementation 
visits across all 26 Round 1 colleges, cross-referenced against intervention reports submitted by colleges to the 
Achieving the Dream partners. 

NOTES: Percentages add to more than 100 because some strategies target more than one student group and are 
thus included in more than one category. More specifically, 10 percent of strategies target both first-year students 
and developmental education students (for example, a success course for first-year students with developmental 
placements), and 4 percent of strategies target both developmental education and gatekeeper students (for 
example, a learning community linking a developmental course with an entry-level college course). 

The “Other” category includes a handful of strategies targeting students of color, low-income students, 
students planning to transfer to four-year institutions, or students with English as a Second Language. 
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Direct Strategies and Achievement Gaps 

While colleges focused their strategies on developmental education and first-year student 
populations, they generally avoided publicly targeting interventions to specific racial, ethnic, or 
economic groups. The handful of exceptions included a small group of colleges that implemented 
mentoring programs specifically for men of color. Twelve schools in total also participated in 
Lumina Foundation’s Dreamkeepers (11 schools) and Angel Fund Emergency Aid (1 school) 
programs'^ — which offered emergency grants or loans to students who faced financial difficul- 
ties that could lead them to drop out of school — but outside of those programs, only a couple 
of colleges reported interventions tailored to students’ financial needs.'”* College leaders 
specified a number of reasons for their decisions not to focus strategies on specific racial or 
economic subgroups. Several felt that other factors, such as developmental education place- 
ment, were more meaningful indicators of student need; others argued that racial and economic 
divisions were not useful given the composition of their particular student bodies (especially at 
institutions where a large majority of students come from low-income families or are students of 
color). Some colleges also reported that they wanted to focus on improving outcomes for all 
students, not just those with certain demographic characteristics. 

Rather than target students based explicitly on race, ethnicity, or income level, some col- 
leges favored subtler approaches, focusing or tailoring strategies to better meet the needs of 
particular student groups within the entire student body. For example, in Alamo Community 
College District, personnel noted that male students of color were not using tutoring services to 
the same extent as other students. The college responded by convening a focus group of African 
American male students to share their thoughts about tutoring. Based on feedback from the focus 
group that some students felt uncomfortable attending tutoring in a centralized location, the 
college expanded its tutoring services online and in various, less formalized locations. In a similar 
vein, most of the Round 1 colleges disaggregated program data, and some actively used these data 
to identify the differential impacts that their strategies might have on subgroups of interest. 

Scale of Direct Strategies 

The ultimate goal of the Achieving the Dream initiative is to improve student outcomes 
institutionwide, but the needle on student achievement cannot be moved through small-scale 



'^Though funding from Dreamkeepers was contingent upon participation in Achieving the Dream (Geck- 
eler, 2008), most of the 1 1 colleges that participated in Dreamkeepers considered this project to be outside the 
scope of Achieving the Dream. 

'“'Santa Fe Community College implemented an innovative scholarship strategy by piloting three alterna- 
tive models of scholarship delivery, while South Texas College focused its efforts on developing labs and 
workshops to improve students’ understanding of financial aid opportunities. 
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pilot projects alone. Instead, colleges must extend promising interventions to a substantial 
percentage of their target populations. For the purpose of this analysis, strategy scale was 
calculated using estimates from college personnel about the approximate number of students 
reached in a given semester (generally fall 2008, depending upon available data), as well as the 
total size of the target population for a given intervention. Strategies were designated “large 
scale” if they reached over 25 percent of their intended target populations, “medium scale” if 
they reached between 10 percent and 25 percent, and “small scale” if they reached under 10 
percent of their target populations. 

While the Round 1 colleges were generally able to commit the resources needed to ex- 
pand a small number of interventions (a majority had either one or two large-scale strategies), 
some found that the reach of their strategy implementation exceeded their capacity for scale-up. 
Figure 5.4 illustrates that even by spring 2009, over half of all direct strategies remained small in 
scale, while less than a third were large scale. This is particularly noteworthy in light of the low 
threshold (25 percent) used to define “large scale,” suggesting that far fewer strategies actually 
reached a majority of targeted students. Large-scale strategies were even less common at the 
subset of colleges with student enrollment over 15,000.'^ The percentage of direct strategies 
reaching large scale at those seven colleges (18 percent) was less than half of that at schools with 
enrollment under 2,000 (41 percent). These findings are not surprising given that larger colleges 
had to expand much further in order to reach a given percentage of their student populations. 

The Trade-off Between Intensity and Scale 

One of the most important findings to emerge from this analysis is the relationship be- 
tween strategy “intensity” — or the amount of contact time students had with a particular 
program or service — and scale-up. Direct strategies were classified as “high intensity” if they 
reached students for 10 or more hours per semester, “medium intensity” if they reached students 
for between 5 and 10 hours, and “low intensity” if they reached students for 5 or fewer hours. 
Intensity is of particular relevance to this analysis in light of the expectation that more intensive 
strategies are more likely to meaningfully affect students’ experiences, and thus are more likely 
to improve their outcomes in observable ways. 

Overall, patterns of implementation across the Round 1 colleges suggest a trade-off be- 
tween the intensity and scale of direct student services and instructional strategies. Figure 5.5 
illustrates that only 31 percent of high-intensity strategies were large scale, compared with 38 
percent of low-intensity strategies. On the flip side, high-intensity strategies were nearly twice 
as likely as low-intensity strategies to remain small scale, with 63 percent of all high-intensity 
interventions reaching under 10 percent of their target populations. These findings highlight a 

'^National Center for Education Statistics (2010). 
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Achieving the Dream: Community Colleges Count 
Figure 5,4 

Proportion of Direct Program and Service Strategies Operating at Small, Medium, and 
Large Scale at the Round 1 Achieving the Dream Colleges, Spring 2009 




SOURCE: Estimates of strategy scale are based on information collected from faculty, staff, and administrators 
during implementation visits in fall 2008 through spring 2009, as well as from the Achieving the Dream 
intervention reports submitted by colleges. 

NOTES: Strategy scale is defined as a percentage of an intervention’s intended target population that is reached 
in a given semester (usually fall 2008, depending upon available data). Strategies are characterized as “small 
scale” if they reached under 10 percent of the students in their target population, “medium scale” if they 
reached between 10 percent and 25 percent, and “large scale” if they reached over 25 percent of their target 
population. A total of 187 direct program and service strategies at aU 26 Round 1 colleges were analyzed. 



more widespread dilemma for community colleges that are seeking to maximize the impact of 
their programs in the face of tight resource constraints: “heavy-touch” interventions allow for 
more contact time with students, but they also tend to be more expensive and difficult to 
implement at scale. 
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Achieving the Dream: Community Colleges Count 
Figure 5.5 



Proportion of Direct Program and Service Strategies Operating at Small, Medium, and 
Large Scale at the Round 1 Achieving the Dream Colleges, hy Strategy Intensity, 

Spring 2009 




I I Small scale 



Medium scale 



H Large scale 



SOURCE: Estimates of strategy scale are based on information collected from faculty, staff, and administrators 
during implementation visits in fall 2008 through spring 2009, as well as from the Achieving the Dream 
intervention reports submitted by colleges. 

NOTES: Strategy scale is defined as a percentage of an intervention’s intended target population that is reached 
in a given semester (usually fall 2008, depending upon available data). Strategies are characterized as “small 
scale” if they reached under 10 percent of the students in their target population, “medium scale” if they 
reached between 10 percent and 25 percent, and “large scale” if they reached over 25 percent of their target 
population. 

“The “High-Intensity” pie on the left represents 95 direct strategies that reached students for more than 10 
hours per semester. 

'’The “Low-Intensity” pie on the right represents 66 direct strategies that reached students for 5 or fewer 
hours per semester. 



This trade-off is supported by findings from a recent study of the costs of implementing 
Achieving the Dream at five of the Round 1 colleges.''^ This study revealed that intervention 
strategies were a major area of investment under the initiative, representing an average expendi- 
ture of $2 million per college. Those costs were driven primarily by the faculty and staff time 
invested in strategy development and implementation, and they were concentrated in the final 
three years of the initiative, when colleges were directing their efforts toward strategy scale-up. 
Thus, more intensive interventions — which generally involve longer faculty or staff contact 



''’Zachry and Coghlan (2010). 
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with students and frequently require more advanced planning — also tend to demand more staff 
resources, increase costs, and consequently pose challenges for scale-up. 

The trade-off between intensity and scale can be further illustrated by examining the 
types of strategies that were most likely to be scaled up. In particular, orientation programs and 
“light-touch” advising programs were the most popular types of low-intensity, large-scale 
strategies across the Round 1 colleges. In fact, nearly half of these colleges had orientation 
programs reaching at least one-fourth of their first-year students. This finding is in keeping with 
the preceding discussion of intensity and affordability, as most orientation programs were short, 
one-time events that could reach large groups of students simultaneously. Moreover, a substantial 
number were built upon existing programs rather than designed and planned from scratch. These 
factors all serve to explain the relatively low per-student cost of planning, implementing, and 
scaling up new student orientations. Similarly, while advising interventions varied significantly 
in their intensity and scale, over a third of them were classified as low intensity and large scale. 

Curricular reform and instmctional strategies tended to be more intensive and less likely 
to reach large numbers of students. In fact, interventions that focused on changes in classroom 
instruction, which reflected only about one-fourth of all direct strategies, accounted for over half 
of the high-intensity, small-scale interventions. More specifically, 18 of 19 learning community 
programs fell into this group, along with approximately half of the 26 curricular reform inter- 
ventions. These patterns provide further evidence for the hypothesis that in-classroom reforms 
typically present additional challenges for implementation and scale-up. 

Despite the manifold challenges of scaling up high-intensity strategies, all but four of 
the Round 1 colleges succeeded in bringing at least one such strategy to scale, contributing to a 
total of 29 “high-intensity, large-scale” strategies. This group of strategies was driven over- 
whelmingly by student success course interventions, which emerged as the only type of high- 
intensity intervention to reach large numbers of students at a majority of colleges. In fact, 17 of 
22 student success courses were large scale, while an additional 3 reached moderate numbers of 
students. Several colleges ensured the broad reach of these strategies by enacting mandates for 
students to take their success courses. While a few homed in on specific groups of students, 
such as those with multiple developmental needs, others required the course for new students. 

The scale-up of other types of intensive interventions was far less consistent. Howev- 
er, as described in Box 5.1, a substantial minority of colleges achieved pockets of success 
scaling up curricular reform strategies, supplemental instruction programs, and instructional 

*^Two additional factors also facilitated the scale-up of orientation programs at a subset of the Round 1 
colleges. The first is that some colleges actually generated money through their orientation programs, as 
increased student enrollments led to higher rates of funding. The second is that a handful of colleges introduced 
mandates for students to attend orientation, thus substantially increasing the reach of these programs. 
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Box 5.1 



Promising Approaches for Scaiing Up Strategies 

Community colleges face a number of challenges in scaling up intensive intervention strat- 
egies, including high per-student costs, staffing shortages, space limitations, scheduling 
demands, and difficulties in recruiting sufficient numbers of faculty, staff, and students to 
participate. Some of the Round 1 colleges surmounted these challenges by relying on in- 
novative approaches to effectively scale up promising strategies. These approaches in- 
clude: 

• Addressing student recruitment concerns by enacting mandates for participa- 
tion. For example, Valencia Community College made student success courses 
mandatory for all students placing into three areas of developmental education, while 
several other colleges required all first-year students to take the course. 

• Addressing stafHng shortages by using administrators and student services staff 
as success course instructors. When success course scale-up and mandates created 
the need for additional sections, a number of colleges turned to administrators and stu- 
dent services staff to teach these courses. With Achieving the Dream funding, some 
colleges were able to provide training to all student success course instructors to en- 
sure consistency in curriculum coverage and delivery. Using a broader array of in- 
structors offers the dual benefits of providing students with an extra opportunity to 
connect with administrators and staff, as well as fostering greater buy-in from admin- 
istrators and staff to a student-centered vision of instruction. 

• Addressing cost limitations by obtaining state funding, as well as developing 
technology and hybrid course approaches. A number of colleges were able to ob- 
tain state and tuition funding for student success courses by aligning the course name 
and number with those of courses already recognized by the state. In the realm of math 
curricular reform, a few colleges sought scalable change by adding a software compo- 
nent to their courses. For example, Galveston College used MyMathLab software to 
address the need for supplemental learning assistance in its developmental math 
courses. This software allows students to complete math assignments while receiving 
help from both the software and the professor while professors are able to monitor 
student progress and address challenges that students are facing with the material. In a 
similar vein. Coastal Bend College initiated computer-assisted math labs to increase 
the number of students passing and progressing through its math courses. Strategies 
such as these are often described as “hybrid” approaches because they combine tech- 
nology, hands-on assistance, and access to an instructor or tutor. In order to provide 
their students with sufficient numbers of qualified tutors, some colleges, such as 
Hillsborough Community College and Wayne Community College, have com- 
bined their computer lab with their tutoring centers, while others, such as Martin 
Community College and Tallahassee Community College, have created systems for 
faculty to spend some of their office hours in the computer lab. 
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reforms as well. One promising approach to scalable curricular reform was to enhance math 
instruction through the use of computer aids. Dona Ana Community College, Coastal Bend 
College, and Galveston Community College each made use of different computer systems — 
EducoSoft, Hawkes, and MyMathLab, respectively — to enhance instmction in their develop- 
mental math courses. Each of these interventions was based on the pairing of regular classroom 
materials with computer-assisted math labs, resulting in substantial curricular changes for all 
students in the affected courses. Outside of computer-based instruction, a small group of 
colleges undertook other curricular reforms designed to improve articulation and alignment 
across course levels or reinforce critical skills such as reading and practical reasoning. Einally, a 
couple of colleges achieved intensive, widespread changes in the classroom by focusing on 
active and collaborative learning approaches. Eor example, Patrick Henry Community College 
has adopted an intensive cooperative learning program, which is described further in the section 
on professional development strategies below. 

Policy Change Strategies 

Though they do not generally involve direct contact among faculty, staff, and students, 
institutional policies are nonetheless an important factor influencing students’ college expe- 
riences and chances of achieving success. Well-formulated policies can ensure that students are 
placed into courses that reflect their abilities and interests, directed toward an efficient pathway 
for meeting their goals, and supported with academic and financial resources. By contrast, 
misguided policies can present unnecessary obstacles that impede or delay students’ progress. 
Many of the Achieving the Dream colleges have recognized the importance of strong institu- 
tional policy as a lever for improving student success across large numbers of students. Approx- 
imately two-th ir ds of the Round 1 colleges undertook at least one policy reform strategy, for a 
total of 30 strategies across the 26 colleges. (See Box 5.2 for examples.) 

Types of Policy Change Strategies 

By a wide margin, the most common policy reforms were designed to improve assess- 
ment and placement procedures to match students with the courses that best served their needs. 
Eor example. University of Mexico-Gallup mandated that all first-year students scoring below a 
certain level on its placement exam be assigned to developmental courses; this college was also 
one of several to adjust its placement test cut-off scores as part of its work under Achieving the 
Dream. Another handful of colleges focused on adopting new placement tools or improving 
existing ones to better assess student preparedness. 

Outside of assessment and placement, the next most popular areas for policy reform were 
in registration and course sequencing. Registration strategies ranged from providing priority 
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Box 5.2 



Types of Policy Change Strategies Implemented Under 
the Auspices of Achieving the Dream 

Assessment and Placement 

• Improvement of assessment tools for incoming students 

• Review and revision of placement test cut-off scores 

• Adoption of mandatory placement for developmental education students 

Registration 

• Priority registration for developmental education students 

• Elimination of late registration based on evidence from data analysis 

• Provision of online registration option 

Course Sequencing and Prerequisites 

• Resequencing of developmental education courses 

• Adoption of prerequisites or mandatory course sequences based on data 

• Reorganization of courses across levels and departments 

Identification and Removal of Barriers to Success 

• Initiative to streamline graduation process and requirements 

• Simplification and improvement of financial aid procedures 

• Adoption or revision of attendance policies 



SOURCE; Information collected from faculty, staff, and administrators during two rounds of im- 
plementation visits across all 26 Round 1 colleges, cross-referenced against intervention reports 
submitted by colleges to the Achieving the Dream partners. 



registration opportunities for targeted groups of developmental education students to eliminat- 
ing late registration based on evidence that students suffered academically when they missed the 
first few sessions of a course. In course sequencing, a handful of colleges implemented new 
policies governing students’ progression through their developmental and credit- bearing 
coursework. Some have also initiated cross-institutional efforts with their local high schools and 
universities to improve curricular alignment across the K-16 spectrum. 
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Target Populations for Policy Change Strategies 

Not surprisingly, developmental education again emerged as the dominant target area 
for policy reforms. Over half of all policy change strategies were focused specifically on 
developmental education, with an emphasis on changes in assessment and placement processes. 
Many of the remaining policy strategies were directed broadly to all students or to those in their 
first year. However, a small but noteworthy handful of policy reforms were designed with an 
eye to reducing achievement gaps across racial and income groups. For example. Central New 
Mexico Community College developed a strategy to improve graduation rates, with a particular 
focus on its Hispanic male students. Its multipronged Graduation Initiative strategy included 
policy reforms to remove structural barriers to graduation, streamline graduation requirements, 
and better identify students who were nearing graduation. South Texas College also aimed to 
improve its achievement gaps, though it took a different approach to equity-driven policy 
reforms, focusing instead on improving its processes for delivering financial aid to low-income 
students. 

Professional Development Strategies 

Besides improving upon the programs, services, and policies that affect students. 
Achieving the Dream colleges are expected to leverage faculty and staff engagement to improve 
student success. Indeed, the initiative highlights the importance of professional development in 
creating more systemic changes in community colleges’ improvement efforts (discussed in 
Chapter 6). Professional development proved to be an important area of college activity under 
Achieving the Dream; every college reported at least one strategy in this area, while a handful 
considered it an important focus of their work under the initiative. Some colleges created new 
centers dedicated to faculty and staff training, while others, such as Southwestern Texas Junior 
College, increased their funding for professional development and h ir ed new directors to 
oversee this work. Findings from an MDRC cost study of Achieving the Dream further high- 
light the centrality of this work: investments in professional development, which averaged 
$654,000 across the five colleges in that study, represented an important share of colleges’ 
spending on activities related to the initiative.'* Box 5.3 captures some of the wide variety of 
professional development strategies that were implemented at the Round 1 colleges under the 
auspices of Achieving the Dream. However, rather than enumerate each of these strategies, this 
section focuses specifically on those professional development strategies that were explicitly 
linked to colleges’ direct strategies under the initiative."’ 



'*Zachry and Coghlan (2010). 

'®Other types of professional development are discussed in Chapter 3 (in reference to training in data anal- 
ysis) and Chapter 4 (in reference to broad engagement). 
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Box 5.3 



Types of Professional Development Strategies Implemented 
Under the Auspices of Achieving the Dream 

One-Time Events on Campus 

• Campus-wide in-services and retreats focused on student success 

• Guest speakers with expertise in specific strategies or institutional reforms 

Conferences 

• Achieving the Dream Strategy Institutes 

• State and regional community college conferences 

• Conferences devoted to developmental instruction or to specific subject areas 

• General funding for faculty and staff to pursue conferences of interest 

Other Travel Opportunities 

• Visits to other colleges to learn about key strategies and accomplishments 

• Evergreen Institute for faculty and staff participating in learning communities^' 

Workshops 

• Sporadic training on select pedagogical approaches 

• Weekly lunch events for faculty and staff 

• Customer service and diversity training for student services staff 

• Advising workshops, particularly for faculty 

• Training for faculty and staff in data collection, analysis, or usage’’ 

Orientation Programs 

• Student success-focused orientation events for new faculty and staff 

Ongoing Development Series 

• Ongoing training related to specific direct strategies or instructional techniques 

• Train-the-trainer models for self-perpetuating professional development 

• Professional development centers hosting ongoing training series 

• Intensive leadership development programs and academies 



(continued) 
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Box 5.3 (continued) 



Support for Faculty Research 

• Mini-grants for independent classroom research projects 

• Data analysis training for pockets of interested faculty^ 

Print and Electronic Resources 

• Resource centers with information about effective approaches to serving students 

• Wehinars and podcasts on specific topics of interest 

• Faculty handbooks 



SOURCE; Information collected from faculty, staff, and administrators during two rounds of im- 
plementation visits across all 26 Round 1 colleges, cross-referenced against intervention reports 
submitted by colleges to the Achieving the Dream partners. 

NOTES: “The Washington Center at Evergreen State College hosts an annual National Summer 
Institute on learning communities, drawing on research and practitioner experience to help col- 
leges initiate or improve upon their learning communities programs. 

’’Professional development related to data analysis and usage is addressed in Chapter 3. 



Professional Development Supporting Direct Strategies 

Over one-third of all professional development strategies under Achieving the Dream 
were designed to improve the implementation of colleges’ direct interventions. For example, the 
majority of those 15 colleges that implemented learning communities also reported some form 
of accompanying training for faculty and staff. Professional development related to advising, 
related to advising, instmctional techniques, curricular reform, and tutoring was also reported at 
a number of colleges. Some colleges sent faculty and staff to off-campus training events, such 
as the multi-day learning community institute at Evergreen State College. Others drew upon the 
expertise of their peers, funding teams of faculty and staff to visit other institutions that had 
excelled in the implementation of a particular strategy. For example, in preparation for restmc- 
turing its own advising system. Coastal Bend College sent members of its advising task force to 
learn about the models being employed at other institutions. Other colleges took similar ap- 
proaches to empower cadres of faculty and staff to become experts in a particular strategy, with 
the expectation that they would share their knowledge and enthusiasm with other faculty and 
staff upon their return. 
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In addition to off-site visits, on-campus “train the trainer” models were employed at a 
handful of the Round 1 colleges. One prominent example of this approach was at Patrick Henry 
Community College, which began by funding a small group of faculty to become certified 
“collaborative learning” trainers who could lead professional development for their peers on 
campus. This model was reinforced by small stipends provided to all faculty, both full time and 
adjunct, who were willing to go through an intensive professional development program. 
Through this approach, Patrick Henry has now provided collaborative learning training to 80 
percent of its faculty. Furthermore, the college has developed a self-sustaining training center — 
called the Southern Center on Active Learning Excellence, or SCALE — to extend collabora- 
tive learning training to faculty from community colleges across the state and beyond (described 
in more detail in Chapter 6). 

In sum, the relative popularity of strategy-specific professional development reflects a 
promising trend: college leaders sought to leverage faculty and staff training in order to fortify 
the implementation of their direct strategies. Not all colleges pursued this type of alignment, nor 
were most direct interventions associated with specialized professional development. However, 
the synergy that emerged between professional development and direct strategy implementation 
at many colleges suggests that college leaders recognized the importance of training personnel 
as a crucial step in improving programs and services. 

Recipients of Professional Development 

Professional development under Achieving the Dream tended to be fairly evenly dis- 
tributed between strategies that were designed to reach a broad swath of campus personnel — 
including both faculty and staff — and strategies that were targeted specifically to faculty 
members. Many efforts in the latter category were focused specifically on subgroups of faculty 
members, such as those in a particular academic department or those teaching developmental 
education courses. However, in keeping with discussions throughout this report, professional 
development reaching adjunct faculty was noticeably rare. Eurthermore, only a handful of 
professional development strategies were targeted specifically and exclusively to student 
services staff. This finding holds in spite of the fact that student support service interventions 
made up nearly half of all direct strategies implemented at the Round 1 colleges. 

Strategies Oriented Toward Externai Stakehoiders 

The final category of strategies presented in this chapter comprises colleges’ efforts to 
engage external stakeholders in their student success agendas. (See Box 5.4 for examples.) 
While less than half of the Round 1 colleges reported strategies in this area, a couple of colleges 
had four or five strategies that were focused on external stakeholders. A significant majority of 
external stakeholder strategies were directed toward strengthening colleges’ relationships with 
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Box 5.4 



Types of Externally Oriented Strategies Implemented Under the 
Auspices of Achieving the Dream 

Collaboration with High Schools 

• Forums for dialogue between eollege and high sehool personnel 

• Cross-institutional task forees foeused on artieulation and alignment 

• Dual enrollment programs and “early eollege” high sehools 

• Early plaeement testing in “feeder” high sehools 

Engagement of Community Members in Efforts to Promote Equity and 
Reduce Achievement Gaps 

• Community conversations focused on promoting success among disadvantaged stu- 
dents 

• Community input into strategic planning 

• Relationship-building between people of color in the community and students 

Communication with Parents and Family Members 

• Parent/family orientation programs, including information about academics and finan- 
cial aid 



SOURCE; Information collected from faculty, staff, and administrators during two rounds of im- 
plementation visits across all 26 Round 1 colleges, cross-referenced against intervention reports 
submitted by colleges to the Achieving the Dream partners. 



local school districts. Several of the Round 1 colleges used their participation in Achieving the 
Dream as a platform to initiate conversations with high school administrators about how to help 
students make the difficult transition into community college. At a subset of these colleges, 
cross-institutional conversations led to the subsequent implementation of targeted interventions 
within feeder school districts,^'’ including dual enrollment programs and early placement testing 
for high school students. Where external engagement efforts were not focused on alignment 
with high schools, some sought to increase community support for their efforts to narrow 



^°“Feeder” school districts are those K-12 school districts that are located near a community college. As a 
result, students from these schools often “feed into” the community college population. 
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achievement gaps across racial and ethnic subgroups of students. For example, a handful of 
colleges hosted community discussions centered on academic success among students of color. 



Colleges’ Use of Data to Inform Strategy Development and 
Refinement 

During their initial year of exploration and planning, Achieving the Dream colleges are 
expected to select intervention strategies based on their analyses of student outcomes data. The 
ideal is that colleges adopt strategies that show promise of addressing priority problems and 
narrowing achievement gaps, and that they draw upon available research to develop well- 
informed plans for implementation. Furthermore, once these strategies have been piloted and 
evaluated (see Chapter 3 for a discussion of strategy evaluation), colleges are expected to make 
informed decisions about their continuation, refining and scaling up those strategies that yield 
promising outcomes while improving or eliminating those that do not. The following discussion 
reveals that the Round 1 colleges made good strides in being more deliberative in their strategy 
development, but most still have work ahead of them to fully utilize data to make decisions 
about these interventions. 

Data and Strategy Development 

MDRC’s baseline research in spring 2006, during the Round 1 colleges’ first year of 
implementation following their planning year, revealed that colleges had mixed success using 
data to develop strategies. On the one hand, they tended to honor the spirit of Achieving the 
Dream by reflecting broadly on priority problems in student performance. About half of the 
Round 1 colleges conducted some form of review of their longitudinal student data to identify 
priority areas for reform, and a majority drew from qualitative focus group data to home in on 
students’ perceived barriers to success.^' Furthermore, as illustrated by the high proportion of 
strategies that were targeted intentionally toward developmental education students and first- 
year students, colleges frequently chose broadly defined focus areas for their strategies that 
related in some way to the problems they identified. 

On the other hand, only about one-third of the colleges were able to draw direct ties be- 
tween their analysis of outcomes data and their selection of specific strategies at baseline. This 
finding is not surprising in light of the fact that most of the Round 1 colleges were still very 
early in their development of a culture of evidence when they began their planning years in fall 
2004.^^ They generally did not have the data capacity to undertake quantitative analyses that 

^‘Brock et al. (2007). 

cited in Chapter 3, only 8 of the original 27 colleges had the data capacity needed to develop a culture 
of evidence when they began in Achieving the Dream. 
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would be sufficiently sophisticated and specialized to yield meaningful information for strategy 
selection. Instead, a few colleges conducted literature reviews during their planning year, but 
they were confronted with a paucity of experimental research documenting effective interven- 
tion strategies at community colleges.^^ The result was that much of the Round 1 colleges’ early 
strategy selection was based on qualitative knowledge of “best practices” gleaned from Achiev- 
ing the Dream conferences, as well as ideas for intervention strategies that predated colleges’ 
participation in the initiative.^"^ In addition, as presented in the discussions of conference 
attendance and professional development in Chapter 4 and this chapter, many colleges relied 
upon the faculty and staff who were charged with leading their interventions to gather know- 
ledge about how to design and implement them. This knowledge generally came from the 
Strategy Institutes, site visits to other colleges, and strategy-specific conferences and trainings. 

Data and Strategy Refinement 

By spring 2009, over one-third of the Round 1 colleges were making effective use of 
student outcomes data to refine strategies. These colleges tended to be above average in their 
institutional research capacities and in the strength of their strategy evaluations, both of which 
facilitated their efforts to gather reliable, strategy-specific data that could be used to improve 
their interventions. For example, one college developed a formal system in which task forces 
continuously reviewed data and made improvements to select strategies. Another achieved a 
similar process of continuous, data-based improvements through its evaluation of a new student 
orientation program. After distributing surveys during the pilot year of this strategy, college 
personnel responded to student feedback by integrating registration activities into the orientation 
sessions. They subsequently expanded and mandated this program based on promising patterns 
of persistence among orientation attendees. A third example of applied strategy evaluation 
occurred when a supplemental instruction program was refined and expanded based on promis- 
ing findings from a simple comparison evaluation. 

Though some colleges had notable accomplishments in strategy evaluation and refine- 
ment — which they attributed in large part to their participation in Achieving the Dream — 
most still demonstrated significant room for improvement. Half of the Round 1 colleges made 
some strategic use of data but fell short in the consistency and rigor of their data usage. For 
example, few reported making decisions about strategy scale-up that were explicitly guided by 
their evaluations. Furthermore, a small handful of colleges showed little or no evidence of 



^^Scrivener and Weiss (2009), p. ix. It was this need for high-quality evidence about strategy effectiveness 
that motivated MDRC’s four random assignment studies under Achieving the Dream. Unfortunately, though, 
the results of these studies were not available to the Round 1 colleges during their early years of participation in 
the initiative. 

^“Brock et al. (2007). 
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applying their evaluation results to strategy refinements, which is not surprising given that all of 
them continued to struggle with the evaluation process itself (as discussed at greater length in 
Chapter 3). 



Summary 

Colleges can expect real changes in student outcomes only when they extend meaning- 
fully improved programs and services to significant numbers of students. In other words, a 
strategy will make an observable impact on institutional performance only if it (1) raises 
academic outcomes among the students who are served, and (2) reaches enough students to 
“move the needle” on collegewide measures. 

The analysis in this chapter reveals that colleges implemented a large number of strate- 
gies representing an enormous investment of time and work by college personnel — an average 
of seven strategies at each college and a total of more than 200 strategies at all 26 colleges. 
They also made noteworthy progress toward scale-up, as nearly all succeeded in expanding at 
least one direct strategy to reach at least 25 percent of its intended target population. However, a 
majority of strategies across the Round 1 colleges remained small in scale, particularly when 
they involved the kind of intensive contact that might be expected to meaningfully influence 
students’ performance. The result is that the benefits of promising interventions were frequently 
extended only to small pockets of students. 

The difficulty that colleges experienced in scaling up strategies raises two important 
questions for the initiative. First, would Achieving the Dream institutions be better served if 
they aimed to implement fewer strategies? The Achieving the Dream model encourages 
colleges to pilot new ideas before evaluating and selecting those that prove to be most effective. 
However, some colleges may have overextended themselves in their planning stages, making it 
difficult for them to improve and expand even their most promising interventions. In the words 
of one Round 1 college president, “Part of the reason we have not accomplished as much as we 
could is that from the beginning, we set out to do too many things.” The college in question 
made important progress in implementing over a dozen direct strategies for policy change and 
professional development, but even those strategies represented only a fraction of the vast 
number of interventions it had envisioned under Achieving the Dream. Overreaching was a 
common experience among the Round 1 colleges, with many that were unable to continue with 
all of their planned strategies throughout their participation in the initiative. They found that 
both staffing and funding were stretched too thin across the multitude of interventions they 
wanted to pursue. 

The second question is focused not on colleges’ approach, but rather on the initiative’s 
model itself: how can the Achieving the Dream partners help colleges to identify effective 
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strategies that can be scaled up at low cost, or else to locate supplemental funding for scale-up? 
The level of funding provided by the initiative was instrumental in jump-starting colleges’ 
strategic work, but it was rarely sufficient to bring their ideas to scale. It will be important for 
the initiative partners and colleges to work together to identify practical approaches and funding 
streams to support this work. 

In conclusion, the colleges’ challenges with strategy scale-up underline the manifold 
demands of Achieving the Dream’s theory of improvement. While the Round 1 colleges have 
made important advances in strategy development and implementation, their stmggles with 
scale-up have important implications for their ability to make headway on the institutional 
performance measures presented in Chapter 8. 
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Chapter 6 

Establishing a Culture of Continuous Improvement 



While strengthening their leadership, data analysis, engagement, and strategy develop- 
ment can provide important levers for improving students’ success, colleges’ ability to institu- 
tionalize such practices is key to creating lasting change. Encapsulating Achieving the Dream’s 
fifth step of the institutional improvement process (see Figure 1.2 in Chapter 1), establishing a 
culture of continuous improvement focuses on helping colleges develop strong management 
and accountability systems to support their multiple efforts to improve students’ achievement. 
In the view of Achieving the Dream, these systems include standing committees to monitor 
colleges’ improvement work and processes for making policy and planning decisions based on 
evidence of students’ achievement (see Box 6.1). Colleges are expected to draft and implement 
a strong strategic plan, which will guide their vision for improving student success and set 
specific benchmarks for monitoring their improvement. The initiative also encourages colleges 
to integrate their student success work with other institutional efforts such as fund-raising, 
accreditation, and budgetary allocations to create more systemic processes for increasing 
students’ achievement. Finally, Achieving the Dream encourages colleges to invest in improv- 
ing the skills of their faculty and staff through professional development opportunities that 
reinforce their efforts to strengthen students’ performance. 

From Achieving the Dream’s perspective, it is this centralization of the change agenda 
that brings a college’s efforts to increase student achievement to full fmition. While other steps 
in the improvement process allow colleges to revise individual practices and programs, this final 
step pushes colleges to develop an integrated system for supporting and managing these 
changes institutionwide. This chapter analyzes how successful the 26 Round 1 colleges were at 
developing such systems and establishing an organizational process to support the integration of 
their vision throughout the college. 

Overall, the implementation findings reveal that the majority of Round 1 colleges had at 
least some institutional systems in place that monitored and used data on student outcomes, 
although the strength of those systems varied across schools. A number of colleges used at least 
some form of data analysis in their strategic planning and budgeting processes, and many tied 
their external grants and accreditation activities to their work under Achieving the Dream. 
Additionally, while the extent of their reach and influence varied, a number of colleges had task 
forces or committees that monitored their efforts to improve student outcomes. Finally, nearly 
all of the Round 1 colleges provided professional development activities to their faculty and 
staff, with just over half providing more intensive professional development services to promote 
their faculty and staff skill-building. 
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Box 6.1 



Principles and Practices for Increasing Student Success 
Through Institutional Improvement: Practices Associated with 
Principle 4, Systemic Institutional Improvement 

Institutional Management 

• College has established a strategic planning process that relies on data to set goals for 
student success and to measure goal attainment. 

• Plans for a given year are driven by a limited set of strategic priorities that have a fo- 
cus on student success. 

• College regularly evaluates its academic programs and student services to determine 
how well they promote student success and how they can be improved. 

• Decisions about budget allocations are based on evidence of program effectiveness 
and linked to plans to increase student success rates. 

• College uses external grant funds strategically to support systemic efforts to improve 
outcomes for students broadly, not just for isolated projects that benefit small numbers 
of students. 

• College works actively to enlarge and sustain pilot programs or practices that prove to 
be effective. 

• Student success agenda is integrated with ongoing accreditation activity, reviews, and 
self-study. 

Organization 

• College has a standing committee or committees that are responsible for guiding and 
monitoring efforts to improve student outcomes. 

• Major meetings, organizational units, and work groups focus regularly on the student 
success agenda. 

Professional Development 

• College offers faculty and staff professional development opportunities that reinforce 
efforts to improve student success and close achievement gaps. 

• College’s induction and orientation activities for new faculty and staff, including ad- 
junct and contingent faculty, foster a commitment to student success. 

• College provides training to faculty and staff on using data and research to improve 
programs and services. 



SOURCE: Field Guide for Improving Student Success (2009), from the Achieving the Dream: 
Community Colleges Count initiative. Available online at 

www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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Colleges’ Development of a Systemic Institutional Improvement 
Process 

Key to Achieving the Dream’s vision for continuous improvement is colleges’ estab- 
lishment of strategic planning and management systems to monitor and coordinate their student 
success efforts (see Box 6.1). In order to institutionalize their earlier efforts, colleges are 
expected to develop a number of committees, task forces, and policies aimed at strengthening 
their reforms and making their changes more concrete. For example, colleges are expected to 
develop standing committees that guide their work toward improving student outcomes, as well 
as a strategic planning process that uses data to set goals for students’ achievement and measure 
their progress. As part of this process, colleges are asked to evaluate their programs and services 
regularly to determine how they can be improved and to make budget allocations to support 
those policies and practices that are most beneficial for increasing students’ success. 

As noted earlier, though Achieving the Dream has codified a number of its practices 
and supports, the Round 1 colleges began their work while the initiative was still under devel- 
opment. Thus, although Box 6. 1 emphasizes a number of practices that colleges are expected to 
emulate, those practices were presented in this iterative form only during the final years of the 
Round 1 colleges’ implementation. Therefore, while this chapter summarizes the Round 1 
colleges’ progress with those tasks, it provides a look (like the chapters before it) at the initia- 
tive’s early influence as it was still being developed. 

Strategic Planning 

Developing a strong strategic plan that sets clear goals for institutions’ reforms and 
benchmarks for measuring their progress is a key element of Achieving the Dream’s vision for 
institutionalizing colleges’ improvement efforts. Nearly half of the Round 1 colleges had a 
strong strategic plan that clearly used data to set goals for student success and allowed colleges 
to measure their attainment of those goals (see Figure 6.1). Many of these colleges also devel- 
oped strong mechanisms for communicating their goals to the college community. For instance, 
a number of schools created institutional scorecards that delineated particular student achieve- 
ment goals (such as targets for student persistence or course pass rates), which the college then 
used to regularly track their progress. Several also had faculty and staff play an active role in 
defining the colleges’ goals each year, either through an integrated committee system that made 
recommendations to the higher-level planning committee or by having faculty submit recom- 
mendations for new programs and policies each year. These efforts helped publicize the 
colleges’ commitment to improve students’ achievement while also integrating the larger 
campus community into this work. 

A little less than half of the Round 1 colleges also had partially developed strategic 
planning systems in place. These systems tended to be less well articulated, and they monitored 
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Achieving the Dream: Community Colleges Count 



Figure 6.1 

Aspects of Systemic Institutional Improvement Processes in Place 
at the Achieving the Dream Round 1 Colleges, Spring 2009 
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SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 

colleges’ progress against specific standards for student success less frequently than those with 
a strong strategic plan. For instance, some colleges had general goals, such as improving 
developmental education, but had not yet attached specific achievement benchmarks to those 
goals. Others did not have well developed data and evaluation processes in place by which to 
set specific targets for student success and measure their progress. Additionally, most of these 
colleges tended to involve higher-level administrators in their strategic planning, and involved 
faculty and staff less often in the process. Only two colleges had no clear strategic plan in place, 
primarily because of a high level of administrator turnover. These findings reveal that most 
Round 1 colleges had made important strides in developing an overall plan to monitor their 
institutional progress, even if a majority of colleges needed to strengthen their efforts further. 
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Organization and Management of Student Success Efforts 

In addition to creating an institutionalized plan for improving students’ achievement, 
colleges in Achieving the Dream were expected to develop mechanisms for carrying out that 
plan. In particular, the initiative argued that colleges should develop committees and task forces 
to oversee their improvement efforts and make decisions about resource allocations that would 
support their reform work (see Box 6.1). As the initiative had hoped, nearly all of the Round 1 
colleges had either existing or newly developed committees tasked with monitoring their efforts 
to improve students’ achievement. In fact, findings from the Achieving the Dream cost study, 
which highlighted the investments of five Round 1 colleges in the initiative, reveal that colleges 
generally invested heavily in their broad institutional reform process (see Box 6.2). Committee 
work, in particular, tended to be a key driver of costs.' 

Just over one-third of the Round 1 colleges had particularly strong management sys- 
tems in place, marked by a sophisticated committee system (as shown in the second set of bars 
in Figure 6.1), which involved numerous faculty and staff at the college and allowed these 
individuals considerable leverage in the colleges’ decision-making. In general, each of these 
schools tended to have several lower-level committees that were charged with monitoring 
specific aspects of the success agenda, such as tracking a subset of the college’s strategies or 
monitoring the college’s overall institutional improvement. These committees then made 
specific programming or policy recommendations to each college’s executive committee and 
thus played a substantial role in moving the college’s agenda forward. Each college’s higher- 
level administrative staff also tended to have standing roles in these committees and vice versa 
(for example, faculty on the executive council) so that ideas and recommendations could be 
translated easily across these different groups. 

About 60 percent of the Round 1 colleges had more limited strategic management sys- 
tems in place. Most of the schools had at least one committee that monitored one or more aspects 
of their student success agenda, such as improving developmental education or tracking the 
success of one or more of their strategies. However, unlike the schools with strong management 
systems, these schools tended not to have an overarching network of committees that brought 
these ideas together, leaving many of these small management groups out of the institution’s 
broader decision-making process. Similarly, most of these schools did not have committees that 
used data regularly to make decisions about programming, and even fewer had functioning 



'Zachry and Coghlan (2010). 
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Box 6.2 



The Costs of Undertaking Institutional Reform 

Improving colleges’ commitment to student success, data analysis, and broad engagement 
can provide an important impetus for increased student achievement. However, substantial 
costs may also be incurred in pursuit of these endeavors. In 2010, MDRC published a case 
study of the investments that five colleges made in Achieving the Dream’s institutional 
improvement process. Through an analysis of these schools’ expenses over their five-year 
participation in the initiative, this study revealed that the colleges took Achieving the 
Dream’s recommendations seriously and made substantial investments in their institutional 
research, leadership, and strategies. In addition, it revealed that the key principles of 
Achieving the Dream were consistent drivers of colleges’ overall reform expenses. 

Colleges’ activities and expenditures have particular relevance for considering how policy- 
makers might support these schools’ continued improvement. Summarized below are some 
key findings that provide a context for colleges’ expenditures on institutional reform. 

• The five colleges spent an average of $6.3 million on their hroad institutional 
reform process. While these monies represented less than 5 percent of their overall 
institutional revenues, they reflect the substantial efforts that colleges made to improve 
their institutions. In general, colleges tended to support much of their work through the 
reallocation of existing employee time. These findings reveal that colleges can support 
many of their reforms internally; however, such choices may present opportunity costs 
as colleges reassign faculty and staff to new positions. 

• The grant of $450,000 that each college received from Lumina provided an im- 
portant impetus for change at all five colleges and was used to leverage invest- 
ments of their own funds and external grants. While colleges supported much of 
their work internally, the flexible monies provided by Lumina helped them invest in 
important aspects of their work, most notably professional development. However, 
colleges spent significantly more in implementing their changes, using their own staff 
resources, internal funds, and external grants to continue their work. 

• Investments across the five colleges revealed that heavier spending is likely to 
occur later in colleges’ institutional improvement process. Colleges tended to 
spend few funds early in their institutional reform process, instead investing most of 
their monies in the final few years of their implementation of Achieving the Dream. 
Such findings reveal that colleges’ scale-up of their institutional changes requires addi- 
tional resources and careful planning. 

• Committee work and professional development tended to he key drivers of ex- 
penditures in leadership management at these five schools, while institutional re- 
search costs tended to he driven by data analysis and staffing. Colleges that fo- 
cused on faculty and staff more heavily also tended to have higher levels of spending. 
This finding reveals that efforts to broadly engage faculty and staff also come with a 
significant price tag and should be considered carefully in colleges’ reform efforts. 

SOURCE: Zachry and Coghlan (2010). 
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data teams to analyze program outcomes on a regular basis.^ Leadership turnover also seemed 
to stall the decision-making process at a number of these schools. Indeed, frequent turnover in 
leadership was a substantial factor at the two colleges that lacked real committee oversight of 
their student success efforts. 

Therefore, in addition to developing strategic planning systems, most Round 1 colleges 
had mechanisms in place for bringing these plans to fruition. However, the majority of Round 1 
schools still needed to establish connections among the numerous, individualized task forces 
that were monitoring their efforts, as this work had not yet become the fully functioning system 
that the initiative had envisioned. 

Use of Data in Budgeting 

As part of their strategic management systems, colleges participating in Achieving the 
Dream were also expected to use the findings from their data analyses and evaluations to make 
decisions about programmatic funding (Box 6.1). Ideally, the initiative hoped that colleges 
would use this research to scale up practices that proved to be most promising while diverting 
funds away from programs that showed little effect on students’ achievement. Findings from the 
implementation research at the 26 Round 1 colleges revealed that nearly 60 percent of them 
used data to at least a moderate degree when making decisions about programming and policies 
(see the third set of bars in Figure 6.1). Just over 30 percent of the Round 1 colleges had 
excelled at this work, developing strong links between their budget allocations process and their 
larger institutional decision-making, often as part of their strategic plan. For instance, Hillsbo- 
rough Community College established a sophisticated online strategic planning process requir- 
ing each academic area to set measureable goals and provide evidence of students’ achievement 
to support their budget requests. Many other colleges with evidence-based budgeting systems 
had similar leanings, regularly using data on improved student achievement to make higher- 
level decisions about programming and policy. 

Another 35 percent of the Round 1 colleges also used some aspects of data analysis in 
their budgeting process; however, their use of evidence tended to be limited to a particular set of 
programs or strategies rather than being integrated with the colleges’ larger financial decision- 
making. For instance, some colleges made decisions about funding special projects or strategies 
based on evidence of student success, but these decisions were generally divorced from the 
colleges’ overall budgeting process. In many cases, colleges had not yet developed their data 
capacity enough to be able to make regular decisions based on evidence of students’ achieve- 



^Achieving the Dream recommends that all colleges participating in the initiative develop a data team, 
consisting of institutional researchers, administrators, faculty, and staff, who are tasked with collecting and 
analyzing data on student outcomes. 
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ment. Finally, a few colleges had a high level of staff turnover, making it difficult for them to 
establish more systematic procedures for budgetary decision-making. 

Just over one-third of the Round 1 colleges rarely used evidence of student success in 
their budgeting processes. The primary limitation for most of these schools was difficulty with 
data analysis, as many had limited institutional research departments that had trouble managing 
the colleges’ multiple data tasks. These issues made it especially challenging for them to adeptly 
use data on students’ achievement to guide their budgeting process. 

Therefore, while many institutions had made important strides in connecting their bud- 
geting to program effectiveness, most Round 1 schools still had only limited mechanisms in 
place to bring promising programs to scale. These findings, together with the challenges that 
colleges had with evaluating and scaling up their strategies, suggest that colleges encountered 
real difficulties in establishing effective mechanisms for validating program effectiveness that 
could be used to guide budgetary decisions. Given these challenges. Achieving the Dream may 
need to identify the key roadblocks that inhibit colleges’ ability to link these efforts and provide 
additional supports to help schools overcome these difficulties. 

Integration with Other Institutional Activities 

As part of colleges’ push to establish their reform efforts institutionwide, the initiative 
has also encouraged participating schools to integrate their Achieving the Dream work with 
other institutional efforts, such as accreditation or endeavors to secure external grants (Box 6.1). 
Nearly all of the Round 1 colleges were successful in bringing together these different strands of 
their work. For instance, a majority of the Round 1 schools tied their Achieving the Dream 
work in with their accreditation activities. In many cases, colleges used Achieving the Dream as 
a framework for establishing the benchmarks and accountability systems that their accreditation 
agencies required for their continued support. For many schools, this work centered around a 
particular Achieving the Dream strategy or activity, which the college sought to further expand 
and monitor as part of its accreditation plans. For instance, Paul D. Camp Community College 
sought to expand its Achieving the Dream work on “distance learning” (courses taken online) as 
it renewed its accreditation through the Southern Association of Colleges and Schools (SACS). 
Similarly, Houston Community College focused on increasing its use of data in institutional 
decision-making when developing its proposal for SACS approval. 

In addition to their accreditation work, virtually all of the Round 1 colleges were also 
successful in obtaining external grants that they then used to support the student success 
endeavors that they had started under Achieving the Dream. Most colleges tended to have one 
or more federal grants that supported their efforts, most commonly Title III, Title V, and TRIO 
grants. However, a number of schools also had grants from other sources, such as foundations 
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or local or state governments. Many of these schools used this supplemental funding to further 
scale up their Achieving the Dream strategies or to sustain other practices that were seen as 
effective in improving students’ success. 

These findings underscore how colleges’ Achieving the Dream efforts can be used 
to interconnect their diverse strategies for institutional improvement. While some colleges 
may have struggled with other aspects of systemic reform, their ability to integrate accredi- 
tation and grant-seeking activities with their Achieving the Dream planning and activities 
reveals a promising mechanism for furthering colleges’ institutionalization of their Achiev- 
ing the Dream work. 

Professional Development 

In order to support the continuous improvement of faculty and staff knowledge, 
Achieving the Dream also recommends that colleges offer ample professional development 
opportunities to hone and increase the skills of faculty and staff (Box 6.1). In particular, the 
initiative hopes that participating colleges will offer professional development activities that 
reinforce their efforts to increase student success and help them learn how to use data effective- 
ly to improve programs and services. 

Overall, most of the Round 1 colleges took seriously the call to provide training to their 
faculty and staff. As noted in Chapter 5, all of the colleges provided at least some sort of 
professional development for their faculty and staff, with those activities being a key focus for 
half of the colleges. Similarly, findings from the Achieving the Dream cost study revealed that 
professional development was a key area of expenditure in the five case study colleges (see Box 
6.2). This was particularly the case for colleges that involved large numbers of faculty and staff 
in such activities.^ 

At schools with more intensive professional developmental offerings, these opportuni- 
ties tended to be directed more concretely toward the colleges’ student success efforts and 
provided continuous support for faculty and staff growth throughout the school year. Typically, 
these schools established semester- or year-long growth plans (as described for Tallahassee and 
Central New Mexico community colleges in Box. 6.3), on-site learning centers for faculty and 
staff, or ongoing training related to specific strategies or interventions (as described for Patrick 
Henry’s program in Box 6.3). While these schools also provided other, more sporadic opportun- 
ities for professional growth, such as one-time workshops or speaker presentations, the key 
practice that set these schools apart was their focus on providing continuous support to their 



^Zachry and Coghlan (2010). 
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Box 6.3 

Intensive Professional Development 

While many colleges offered only sporadic training or workshops for faculty and staff, a 
number of institutions developed model programs to continuously support the develop- 
ment of their faculty and staff A few of the most promising programs are highlighted here. 

Patrick Henry Community College 

In order to support the scale-up of its coUahorative learning strategy, Patrick Henry devel- 
oped an ongoing training center, called the Southern Center for Active Learning Excel- 
lence, or SCALE. SCALE provides training in coUahorative learning to both full-time and 
part-time college faculty, including introductory training for novices as well as ongoing 
support for faculty who have already been trained. The center provides a number of sup- 
ports to promote collaborative learning in the classroom, such as free courses for faculty 
and staff to learn about different elements of active learning, coaching and mentoring op- 
portunities, examples of classroom lessons, surveys for students, and training institutes for 
other colleges to learn about collaborative learning. Eaculty leaders direct the center and 
have been certified as collaborative learning trainers through the Cooperative Learning 
Center at the University of Minnesota. This “train-the-trainer” model allows Patrick Henry 
to train increasing numbers of faculty each year while controlling the costs for these ser- 
vices. To date, 85 percent of Patrick Henry’s full-time and 60 percent of its adjunct faculty 
have been trained in the use of collaborative learning in the classroom. 

Tallahassee Community College 

TaUahassee operates a systematic, year-long professional development program for its fac- 
ulty. New faculty members participate in ongoing New Eaculty Seminars through the insti- 
tution’s Center for Teaching Excellence while seasoned faculty members are responsible 
for completing Professional Growth Plans, in which they list goals for increasing student 
success inside the classroom and propose methods for building their own skills and know- 
ledge to make these targets a reality. In addition, faculty members monitor their students’ 
performance outcomes, examine whether and how their target goals have been met, and 
provide justification for their work at the end of each year. Though not always directly 
related to its work in Achieving the Dream, Tallahassee’s ongoing system for faculty de- 
velopment represents a key avenue for colleges’ further development of their faculty’s 
skills and integration of new reforms. 

Central New Mexico Community College 

Like Tallahassee, Central New Mexico also provides ongoing support to its faculty, 
through a semester-long instmctor development course. This course offers faculty an op- 
portunity to focus on topics such as student success, classroom management, and learning 
styles. It teaches faculty members how to recognize students who have disengaged and 
how to apply learning techniques to reintegrate them into the classroom. In addition, this 
semester-long course familiarizes faculty with institutional resources that are available to 
support their instruction, thereby helping new faculty to become acquainted with the col- 
lege. While this professional development course tended to be separate from the college’s 
student success initiatives, it could easily be modeled to provide more sustained training 
and reflection for faculty as part of colleges’ implementation of new reforms. 
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faculty and staff to learn a new skill, pedagogy, or practice that was directly linked with their 
efforts to improve student achievement. 

At the remaining half of Round 1 colleges, professional development tended to be more 
sporadic. While these colleges had some professional development opportunities, they tended to 
be limited to a small subset of strategies, sporadic speakers, or occasional visits to conferences 
or workshops. However, only a few Round 1 colleges did not see professional development as 
an important priority. The reasons for their decreased attention tended to center around budget 
issues or a lack of focus on their Achieving the Dream work in general. 



The Influence of Achieving the Dream on Colleges’ Systemic 
Institutional Improvement Processes 

Achieving the Dream had an important influence on most of the Round 1 colleges’ sys- 
temic institutional improvement processes, with the initiative influencing over 80 percent of 
these schools to at least a moderate degree (see Figure 6.2). For some schools. Achieving the 
Dream brought about a strong shift in focus, moving them more toward an evidence-based, 
student success-driven management process. Achieving the Dream also helped a number of 
colleges refine their systems for monitoring student success, such as developing institutional 
scorecards for measuring progress toward their institutional goals or further focusing their 
strategic planning on evidence-based practices for increasing student success. Additionally, a 
number of colleges attributed their revised committee stmctures to their work in Achieving the 
Dream. Indeed, only four colleges felt no real effects from Achieving the Dream on their 
strategic planning and management. These schools tended to have weaker strategic manage- 
ment systems in general, and were continuing to work at developing more coordinated systems 
to oversee their various projects. 

Similarly, over two-thirds of the Round 1 colleges noted that they devoted a significant 
proportion of their Achieving the Dream funding to their professional development activities. 
Many reported that their Achieving the Dream grants were integral to this work, with several 
colleges using this grant as their central mechanism for funding professional development. Those 
that depended most heavily on their Achieving the Dream grants tended to be smaller colleges 
with limited financial supports. While larger colleges also found the grant helpful, they were also 
able to use their institutional funds to support faculty and staff professional development. 

While Achieving the Dream had an important influence on nearly all of the Round 1 
colleges, the initiative was only one of a multitude of forces pushing many colleges toward 
more systematic management and monitoring. For instance, a majority of schools discussed 
their accreditation process or their presidents’ own vision as driving forces behind their man- 
agement and planning systems. Others noted that they had made only modest changes to their 
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Achieving the Dream: Community Colleges Count 
Figure 6.2 

Influence of Achieving the Dream on Systemic Institutional Improvement Processes 

in Round I Colleges 




SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives. 
Achieving the Dream coaches, and data facilitators. 

management structures as a result of Achieving the Dream, reporting that some of those 
changes were already in progress when they began in the initiative. Such comments reveal that 
Achieving the Dream was one of a number of factors pushing colleges to monitor and improve 
their students’ achievement. 



Summary 

Overall, the Round 1 colleges made important strides in developing more systemic ef- 
forts to increase students’ achievement. A majority of the Round 1 colleges had strategic 
planning mechanisms in place, which established goals for their work, and had established task 
forces to monitor those efforts and take actionable steps to reahze their goals. Similarly, most 
schools offered some form of professional development for their faculty and staff, and they had 



108 



integrated their Achieving the Dream work with other institutional reform efforts, such as 
accreditation and fund-raising. However, these systems needed further strengthening at a 
number of schools, as they were limited in their development of actionable goals and bench- 
marks for student improvement, and in their use of data to monitor their efforts. Additionally, 
they had yet to effectively connect their funding strategies with evidence of program effective- 
ness, making it more difficult to successfully scale up promising programmatic interventions. 
Such findings reveal that Achieving the Dream schools may need additional time to institu- 
tionalize their reform practices. 

Given these challenges. Achieving the Dream might need to give participating colleges 
more time to fully integrate their reforms within the larger structure of the college and streng- 
then those capacities that may be hindering the development of more data-driven systems. As 
noted in previous chapters, most colleges were still heavily engaged in strategy development 
and learning how to evaluate the effects of their reforms on students’ achievement. Similarly, a 
number of schools were still stmggling to ramp up their institutional research capacity to meet 
the demands of the initiative. As such. Achieving the Dream might look toward helping 
colleges take on more sophisticated data analyses that would allow them to build the infrastmc- 
ture to use data regularly within their strategic management and budgeting processes. 

Achieving the Dream might also encourage colleges to develop more systematized pro- 
fessional development opportunities for their faculty and staff that support their efforts to 
continuously improve student achievement. While conferences and workshops may help expose 
faculty and staff to new ideas, more concrete efforts are needed to help these individuals bring 
such reforms into their everyday practice. The systems developed by Patrick Henry, Tallahas- 
see, and Central New Mexico community colleges represent three models for providing more 
sustained growth and development of faculty and staff skills. Achieving the Dream might look 
to efforts such as these to further encourage these individuals’ engagement in colleges’ reforms 
and provide an even stronger foundation for colleges’ efforts to improve students’ achievement. 

Finally, as colleges seek to implement these broader changes, they are also likely to 
spend greater amounts of institutional funds. Findings from the Achieving the Dream cost study 
reveal that colleges are much more likely to spend high levels of institutional funds should they 
seek to develop the types of intensive professional development and engagement stmctures 
recommended by the initiative. Given the likelihood that colleges will invest heavily in this 
work. Achieving the Dream might wish to help them consider how to balance these expenses 
with their other existing commitments and priorities. Providing sound advice that allows 
colleges to plan for their investments and consider the longer-term impact of their reforms may 
help these schools to sustain the changes that they are making on the ground today. 
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Chapter 7 



Institutionalizing Change: 
Progress in Building a Culture of Evidence 



Colleges’ ability to adopt strong leadership, data analysis, and engagement practices 
that are aimed at improving students’ success and their ability to integrate those efforts into their 
institutional work are considered cornerstones of Achieving the Dream’s vision for building a 
culture of evidence. Colleges that are able to build systems “in which data and inquiry drive 
broad-based institutional efforts to close achievement gaps and improve student outcomes” are 
considered to have excelled at these tasks.' Thus, while the previous chapters have emphasized 
colleges’ development of particular sets of practices. Achieving the Dream’s model focuses on 
colleges’ integration of those practices into a coherent and continuous system for institutional 
improvement aimed at increasing students’ achievement. 

Given Achieving the Dream’s relentless focus on strengthening colleges’ data-based 
decision-making, this chapter provides a concluding, summative look at the Round 1 colleges’ 
success in building an overall culture of evidence during their five-year implementation of 
Achieving the Dream. The colleges’ progress in building a culture of evidence is based on the 
strength of their implementation of each of the five steps in the initiative’s continuous im- 
provement model: (1) securing leadership commitment; (2) using data to prioritize actions; (3) 
engaging stakeholders; (4) implementing, evaluating, and improving intervention strategies; and 
(5) establishing a culture of continuous improvement. Thus, each of these factors is taken into 
consideration when evaluating the strength of colleges’ evidence-based decision-making. 

When examining colleges’ overall development of a culture of evidence, the implemen- 
tation research revealed that many schools had made substantial progress in building a culture 
of evidence during their five-year participation in Achieving the Dream. Approximately 81 
percent (21 of 26) of the Round 1 colleges had improved their data-based decision-making by 
spring 2009, nearly double the number of schools that were making progress in spring 2006. 
Only five colleges (19 percent) continued to have major challenges implementing Achieving the 
Dream’s model for institutional improvement, which represents an important decrease from the 
15 colleges (56 percent) that were struggling in 2006. Overall, most of the Round 1 colleges 
made improvements in their data analyses and institutional research capacities while also 
developing a number of interventions to improve students’ achievement. However, areas of 
improvement remained for a number of schools, including the broad engagement of both full- 



'Achieving the Dream (2009), p. 3. 
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time and part-time faculty and the strengthening of colleges’ use of data to make broad-based 
decisions about institutional policies and programming. 



Colleges’ Overall Progress in Developing a Culture of Evidence 

As noted in the previous chapters, a number of colleges made great improvements to 
their data analysis, faculty engagement, strategies, and strategic management practices during 
the course of their participation in Achieving the Dream. As a result, many also made great 
strides in building a culture of evidence during their five years in the initiative. 

Strong Colleges: Colleges with an Institutionalized Culture of Evidence 

As Table 7.1 reveals, 42 percent (11) of the Round 1 colleges had implemented most of 
the practices associated with a strong culture of evidence by the end of their five-year participa- 
tion in Achieving the Dream. These colleges had excelled at building or further strengthening 
all five aspects of Achieving the Dream’s institutional improvement model and (1) had strong 
leadership systems committed to improving student success; (2) had well-developed institution- 
al research departments that conducted intensive, in-depth analyses of their students’ achieve- 
ment; (3) involved faculty, staff, students, and external stakeholders in multiple aspects of their 
reform agenda; (4) had well-developed strategies, with one or more strategies reaching a large 
proportion of their target population; and (5) had instituted multiple processes for strategic 
planning, program review, and professional development, and had made financial and pro- 
grammatic decisions based on evidence of improved student achievement. 

While each of these colleges was strong, many had begun Achieving the Dream with 
well-developed data analysis systems and a strong focus on improving student success. As can 
be seen in Table 7.1, just over half (six) of these strong colleges had exhibited these traits early 
on in the Achieving the Dream process. Despite this trend, a number of these schools credited 
the initiative with helping them develop even better systems for improving student success (see 
Table 7.2). For instance, Valencia Community College, which had started Achieving the Dream 
with a stellar institutional research department and highly engaged faculty and staff, used the 
initiative to catalyze its reform agenda and bring together the college’s four campuses to create 
an institutionwide movement. Similarly, South Texas College, which began the initiative with a 
strong institutional research department, chose to further develop and expand its data analysis 
capacity, hiring three additional institutional researchers and breaking its department into 
smaller divisions to make it easier to share the results of data analyses throughout the school. 

Achieving the Dream had a particularly dramatic influence on helping four of the re- 
maining strong colleges build more evidence-based systems (see Table 7.2 and Box 7.1). These 
“star” schools began Achieving the Dream with a weak or very weak culture of evidence, with 
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Achieving the Dream: Community Colleges Count 
Table 7.1 



Culture of Evidence at Achieving the Dream Round 1 Colleges, 
Spring 2006 and Spring 2009 





Spring 2006 


Spring 2009 


Culture of Evidence Rating 


Number of 
Round 1 Colleges 


Percentage of 
Round 1 Colleges 


Number of 
Round 1 Colleges 


Percentage of 
Round 1 Colleges 


Strong or very strong 
culture of evidence 


6 


23 


11 


42 


Some culture of evidence 


5 


19 


10 


38 


Weak or very weak 
culture of evidence 


15 


58 


5 


19 


Total 


26 


100 


26 


100 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of implementation 
research across all 26 Round 1 colleges, cross-referenced against reports submitted by college representatives, 
Achieving the Dream coaches, and data facilitators. 

NOTE: Percentages may not add to 100 because of rounding. 



few data systems in place and less organized processes for making decisions about student 
success. As such, these colleges tended not to make planning, programmatic, or budgetary 
decisions based on evidence of what works for students, and relied instead on anecdotes or 
hunches when implementing reforms. However, all of these schools noted that Achieving the 
Dream had “made possible a... paradigm shift” that allowed them to “transform” their institu- 
tional research capabilities, “reshape” their strategic planning processes, and focus on “data that 
tells about student success.” These colleges developed in a number of similar ways, including 
making strides in incorporating their faculty, staff, and other stakeholders into their student 
success work and focusing strongly on professional development, which helped integrate their 
faculty and staff into their reform agenda. 

While Achieving the Dream played an important role in these colleges’ development 
and provided some integral supports for their work, these schools seemed to focus more often 
on how the initiative’s theory had helped them build upon the already strong passion for helping 
students that their faculty, staff, and administrators exhibited. Achieving the Dream data 
facilitators played an important role at a few of these colleges, helping them develop systemized 
ways of analyzing and sharing data institutionwide. Additionally, coaches and the Achieving 
the Dream meetings, such as the Strategy Institute, were generally viewed as “very helpful.” 
However, what seemed to make the most difference for development at these colleges were 
strong, visionary leaders at multiple levels of the institution (for example, administrators as well 
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Achieving the Dream: Community Colleges Count 
Table 7.2 



Influence of Achieving the Dream on the Culture of Evidence at Round 1 
Colleges and Progress Made from Spring 2006 to Spring 2009 



Number of Colleges 


Spring 2006 
Culture of 
Evidence^ 


Spring 2009 
Culture of 
Evidence 


Progress Made 
from Spring 2006 
to Spring 2009*’ 


Heavv influence: 8 colleges 


3 colleges 


Strong 


Very strong 


H-1 


1 college 


Some 


Strong 


H-1 


3 colleges 


Weak 


Strong 


+2 


1 college 


Very weak 


Strong 


+3 


Some influence: 11 colleges 


2 colleges 


Strong 


Very strong 


+ 1 


1 college 


Very weak 


Some 


+2 


6 colleges 


Weak 


Some 


+1 


2 colleges 


Very weak 


Weak 


+1 


Little influence: 7 colleges 


1 college 


Strong 


Very strong 


+ 1 


1 college 


Weak 


Some 


+1 


2 colleges 


Some 


Some 


0 


1 college 


Very weak 


Very weak 


0 


2 colleges 


Some 


Weak 


-1 



SOURCE: Information collected from faculty, staff, and administrators during two rounds of 
implementation research across all 26 Round 1 colleges, cross-referenced against reports submitted 
by college representatives. Achieving the Dream coaches, and data facilitators. 

NOTES: While some of the colleges in the "Little influence" category made progress in developing 
their cultures of evidence, they did not consider Achieving the Dream to be a primary motivator for 
these changes. 

‘‘Spring 2006 culture of evidence ratings are based on implementation findings from MDRC and 
CCRC's baseline report on the Round 1 Achieving the Dream colleges (Brock et al., 2007, pp. 96- 
97). 

’’Progress ratings reflect changes in colleges' cultures of evidence between spring 2006 and spring 
2009, as measured by movement across the following rating scale: very weak, weak, some, strong, 
very strong. Eor example, a college that received a "very weak" rating in spring 2006 and a "very 
strong" rating in spring 2009 would receive a progress rating of "h- 4," indicating that it had 
progressed 4 levels in the rating scale. 



114 







Box 7.1 



“Star” Colleges in Achieving the Dream: Institutions that Made 
Solid Progress in Building a Culture of Evidence 

Four of the 26 Round 1 colleges stood out because they had a weak culture of evidence at 

the start of the initiative and made great strides in building up their culture over five years. 

These colleges made changes that set them apart from the other colleges that made less 

progress, including: 

• Multiple senior administrators who were deeply involved in the colleges’ reforms. 

Each college had a cadre of senior leaders, including the vice presidents of instruction 
and student services and the head of institutional research (IR), leading its Achieving 
the Dream work. Two colleges also had presidents who were very involved. 

• Active IR departments that produced accessible reports on students’ achieve- 
ment. Each school focused sharply on increasing IR capacity by hiring new IR staff 
and upgrading its data management system, which allowed it to better manage multiple 
IR demands. Each school also produced institutional- or department-specific reports on 
students’ success, with clearly delineated target goals for students’ achievement. 

• Regular evaluations of their interventions to improve student success. Each col- 
lege systematically tracked its interventions, often comparing students who were par- 
ticipating in a particular program with nonparticipants. These findings were dissemi- 
nated widely throughout the school, with a number of faculty and staff commenting on 
the results. 

• Strong faculty and staff leaders who played an active role in leading the colleges’ 
reform s and interventions. Each of the four colleges had strong participation from 
faculty and staff, who were involved in leadership and policymaking committees at 
their colleges and led the implementation of numerous strategies that were aimed at 
improving students’ success. 

• Integrated committee structures that allowed for regular communication among 
administrators, faculty, and staff. Each of the four schools developed numerous 
standing committees to monitor its reform efforts. These committees brought faculty, 
staff, and administrators together through regular communication, providing a clear 
and consistent voice in their colleges’ larger policy- and institutional decision-making. 

• Consistent attention to financial supports for strategy scale-up and numerous 
strategies operating at scale. Each of these colleges developed systematic ways to 
support broad-scale reforms, often by using federal or state grants to support the addi- 
tional costs. All four colleges had strategies that were reaching substantial proportions 
of their remedial students or students who were in college for the first time. 

• Heavy investment in professional development. Each of the four schools dedicated 
significant funds to training faculty and staff 
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as faculty and other staff) who were able to build well-integrated, multidimensional processes 
for developing and making decisions about their programs and policies. While Achieving the 
Dream was a “galvanizing effort” to help them focus on students’ success, all of these schools 
used Achieving the Dream’s process to actualize the existing strengths within their schools and 
identify the weaknesses, which they then focused on improving. Therefore, while Achieving the 
Dream’s services were useful, a strong, broad-based effort that focused on increasing student 
success on the ground at these colleges was what allowed their Achieving the Dream-inspired 
reforms to take root. 

Colleges with Some Culture of Evidence 

While several colleges made outstanding progress in building a culture of evidence over 
their five years in Achieving the Dream, a number of other schools had also made important 
strides in furthering their evidence-based decision-making. As can be seen in Table 7.1, 38 
percent (10) of the Round 1 colleges had instituted some of the aspects of Achieving the 
Dream’s suggested improvement process in their schools after five years in the initiative. All 
had built (or started with) relatively capable institutional research departments, and most 
schools undertook longitudinal analyses of student outcomes and tracked the success of at least 
some of their student success interventions. Other stakeholders, such as faculty and staff, also 
reported increased awareness of student achievement or had some exposure to reports on 
students’ success in particular interventions. Faculty and staff were also generally involved in 
some aspects of the reform agenda at many of these schools, most often around implementing 
the colleges’ student success strategies, although several were also involved in larger leadership 
committees. Finally, most colleges had some form of strategic planning process in place, which 
at times used student achievement data to inform decision-making. 

Although these colleges had made important strides in engaging with data, several issues 
hindered their ability to develop a more broad-based culture of evidence. While most had 
increased their institutional research departments, half of these schools still suffered from a lack 
of capacity to undertake analyses that are important for documenting students’ progress and had 
challenges with managing the increased requests for student outcomes data. Colleges with very 
small institutional research departments, staffed with only one or two people, had a particularly 
difficult time getting data out to their faculty and staff, often because they were charged with 
conducting multiple analyses at once.^ In addition, some colleges were unable to pursue longitu- 
dinal analyses of their students’ achievement because their information technology (IT) systems 
were limited or weak. For example, one college had no way of tracking its students’ progress 



^In comparison, colleges with a strong culture of evidence tended to have five or more well-trained re- 
searchers leading their data analyses. 
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over multiple years and had no plans for developing such a system, arguing that the cost of 
managing this type of data warehouse was “prohibitive” for a larger institution of its size. 

These colleges also faced important challenges with sustaining the leadership and mo- 
mentum of their work over the course of five years. For instance, more than half of these 
schools faced issues with leadership turnover, which made it difficult to systematically advance 
their improvement agendas. Additionally, the majority of these colleges had leadership teams 
that either didn’t share data across the campus or did not systematically use data to make 
programming and policy decisions. Finally, nearly all of the colleges had faculty or staff who 
were resistant to using student outcomes data or who were not sure how these data should affect 
or inform their everyday practice. Indeed, more than half of these colleges had entire academic 
or student services departments that were either not involved in the college’s Achieving the 
Dream agenda or were far less involved than other personnel. 

While many schools still struggled with important aspects of Achieving the Dream’s 
model, most of them had made progress since spring 2006, with many attributing their devel- 
opment to their work in Achieving the Dream. As can be seen in Table 7.2, 1 1 out of 26 (42 
percent) of the Round 1 colleges said that Achieving the Dream had some influence on their 
development of a culture of evidence. Some of these efforts were centered on an increase in 
their data capacity, through the addition of new staff, upgraded IT systems, or more sophisti- 
cated analyses of their students’ achievement. However, more often Achieving the Dream had 
supported the colleges’ implementation of student success strategies. Many of these schools 
noted that the grant of $450,000 from Achieving the Dream played an important role in this 
process, with some of these colleges observing that the grant was one of the most important 
contributions Achieving the Dream made to their schools.^ Nearly all of these schools used the 
grant to support key aspects of their reform agendas, including the implementation of strategies, 
professional development for faculty and staff, and increasing their IR capacities. 

While most of the colleges with some culture of evidence attributed a number of their 
modifications to Achieving the Dream, most also noted other factors that were important in 
their development. For instance, many of the schools discussed how their accreditation 
processes or grants other than the one provided by Achieving the Dream, such as federal Title V 
or TRIO grants, were the main drivers of the reforms at their colleges. Achieving the Dream 
was seen as “complementing” or “reinforcing” this agenda, but not necessarily as the primary 
inspiration behind the colleges’ efforts to improve student achievement or to use data. Addition- 
ally, a few colleges said that the motivation for their reforms stemmed from an internal push to 
increase student achievement. Similar to colleges with a strong culture of evidence, this internal 



^All of the schools that saw the grant as very important were smaller schools with fewer than 5,000 stu- 
dents. 
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push came more often than not from a strong leader who had a pre-existing commitment to 
improving student success. Thus, while Achieving the Dream had some effect on the culture of 
evidence at these schools, this influence tended to be balanced with other internal or external 
forces pushing these institutions in this same direction. 

Colleges with a Weak Culture of Evidence 

Relatively few colleges were still having major problems developing a culture of evi- 
dence at their schools by the end of their demonstration phase in Achieving the Dream. As can 
be seen in Table 7.1, only five (19 percent) of the Round 1 colleges were still stmggling to 
implement a number of Achieving the Dream’s recommended practices by spring 2009. Very 
weak institutional research departments were the primary difficulty hindering most of these 
schools’ ability to institute a broad, data-driven culture. Most of these schools did not employ an 
institutional researcher continuously during their time in the initiative, relying instead on outside 
consultants or other, less well-trained staff to fill these positions. Sometimes these challenges 
were the result of a poor economic situation, which kept the college from being able to attract 
good technical support. Some schools also had difficulty implementing Achieving the Dream 
across multiple campuses and providing sustained leadership to bring these disparate groups 
together. 

Without strong institutional research capacity, most of these schools were unable to un- 
dertake even the most basic data analyses during much of their time in the initiative. For 
instance, most lacked sophisticated IT systems to manage large quantities of data and thus were 
unable to extract information on student outcomes efficiently. Perhaps as a result of these 
difficulties, few undertook longitudinal data analyses during their time in Achieving the Dream, 
with some colleges hiring outside consultants to develop and submit their data to Achieving the 
Dream’s database. Those that did analyze longitudinal data tended to rely on the reports they 
received from JBL Associates, which provided schools with individual analyses of the student 
outcomes data they submitted to the Achieving the Dream database. Given these challenges, it 
is perhaps not surprising that none of these schools systematically engaged faculty and staff in 
data analysis, nor did they have strategic planning systems that regularly used data on students’ 
success. Furthermore, few of these schools systematically evaluated their strategy interventions 
and made decisions about their improvements based on evidence of promising trends in student 
achievement. 

The challenges facing colleges with a weak culture of evidence tended to be the result 
of forces outside of Achieving the Dream. For instance, all five colleges that still had a weak 
culture of evidence by spring 2009 had experienced a high level of turnover in their leadership 
and administration, making it difficult to sustain any reform efforts at their schools. Additional- 
ly, some of these schools were located in economically challenged areas, making it difficult to 
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attract and pay high-quality institutional research staff. However, at times, these colleges 
were also limited by their interest in, engagement with, or even understanding of Achieving 
the Dream. For instance, a few of these schools initially misunderstood the purpose of 
Achieving the Dream and had focused primarily on strategy implementation. Additionally, 
some had faculty or staff who were resistant to using data or engaging with Achieving the 
Dream altogether. 

While these colleges struggled with many aspects of Achieving the Dream’s model, all 
had also seen some positive results from their involvement in the initiative. Nearly all of the 
schools reported that Achieving the Dream had made them more aware of the importance of 
data, and most had made modest strides in building up their data analysis processes. For 
instance, a few of these schools collected qualitative data regularly through surveys or focus 
groups and used the results to better understand how their students were experiencing college. 
Additionally, a few of the schools were focused on analyzing data (often from JBL’s analyses 
of the Achieving the Dream database) for different student subgroups and noted how those data 
had informed the focus of some of their strategies. Finally, all of these schools had developed at 
least one strategy aimed at increased student success, and some had successfully brought those 
strategies to scale within their institutions. 

While these schools had experienced some positive results from their participation in 
Achieving the Dream, important questions remained about whether they would sustain their 
efforts after completing the demonstration phase of the initiative. Several of these schools noted 
that their Achieving the Dream grants had played an important role in helping them implement 
their student success reforms and that they did not have a plan for how these reforms would be 
sustained after the grant period ended. Unfortunately, it appears that the tenuous culture of 
evidence at some of these schools was built upon funding from the initiative and had not been 
integrated fully within their institutions. Thus, it remains to be seen how much of their momen- 
tum can be sustained after this funding ends. 



Summary 

Many colleges made encouraging progress in further developing institutionalized sys- 
tems for improving student success and improving their culture of evidence during their five 
years in Achieving the Dream. Most Round 1 colleges had developed systems that used data on 
student outcomes to help guide their planning and policy decisions. The majority of colleges 
also had task forces overseeing their institutional improvement efforts. Additionally, many had 
tied their Achieving the Dream work in with other external grants and accreditation activities to 
create a more wide-ranging student success effort. Finally, most colleges had devoted substan- 
tial resources to the training and professional development of their faculty and staff. 
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In addition, nearly all of the Round 1 colleges strengthened their overall culture of evi- 
dence by making substantial upgrades to their data capacity and communication about students’ 
achievement. Twenty-one (81 percent) of the 26 Round 1 colleges had made important progress 
in building a stronger culture of evidence by spring 2009, with only five colleges (19 percent) 
continuing to have major challenges implementing Achieving the Dream’s model for institu- 
tional improvement. However, areas of improvement remained for a number of schools, 
including the broad engagement of both full-time and part-time faculty and the strengthening of 
colleges’ management systems for improving student success. Additionally, few schools were 
using data systematically to monitor the effect of their strategies or their overall institutional 
performance, thus falling short of the initiative’s hope for institutionalized evidence-based 
decision-making. 

While many of the Round 1 colleges made good progress in strengthening their culture 
of evidence, a crucial aspect of the Achieving the Dream model is the improvement of students’ 
achievement. The initiative expects that colleges’ institutional changes and reforms will lead to 
improvements in overall student performance, such as students’ progress through developmen- 
tal education, completion of gatekeeper courses, and their persistence from term to term and 
year to year. Also important is whether or not colleges were able to reduce the achievement 
gaps among various student subgroups. The next chapter of this report turns to examining these 
trends in student performance and what changes can be seen across Achieving the Dream’s five 
key indicators of student performance. 
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Chapter 8 

Trends in Academic Performance 



The previous chapters examined how well the Round 1 colleges implemented the 
Achieving the Dream model for institutional improvement. The initiative expects that colleges’ 
rigorous implementation of the five-step model that is illustrated in Chapter 1 will ultimately 
lead to measureable improvements in students’ academic achievement. That is, the initiative 
anticipates that colleges’ increased attention to student outcomes data and changes to policies 
and practices will lead in turn to measurable improvements in the rates in which students: ' 

1. Complete remedial or developmental courses and progress to credit-bearing 
courses. 

2. Enroll in and complete college-level “gatekeeper” courses. 

3. Complete courses, with a grade of “C” or higher.^ 

4. Reenroll term-to-term and year-to-year. 

5. Earn certificates or degrees. 

As noted in the previous chapters. Achieving the Dream helped many colleges build a 
stronger foundation for accomplishing these goals. College leaders tended to have a strong 
commitment to changing programs and policies to increase students’ achievement, and a 
number of Round 1 colleges made improvements to their data capacity and the dissemination of 
student outcomes data, both of which helped colleges strengthen their culture of evidence. 
Einally, most colleges developed multiple strategies aimed at increasing students’ achievement 
and involved at least some faculty and staff in this reform process. 

While Achieving the Dream can count these changes as successes, many of the Round 
1 institutions were still facing important challenges in 2009 in scaling up those reforms that 
might directly affect students’ achievement. Eor instance, many colleges struggled to undertake 
certain forms of data analysis, such as strategy evaluations, making it difficult to know whether 
and how their strategies might be influencing students’ success. The majority of colleges’ 
strategies also tended to remain small in scale, affecting only selected pockets of the student 

'Achieving the Dream (2009). 

^While the benchmark specifies passing with a grade of “C” or higher. Achieving the Dream colleges did 
not start submitting data on course grades with this limitation until 2007. As a result, all of the course comple- 
tion measures in this report are based on passing courses with a grade of “D” or higher. Completion of de- 
velopmental and gatekeeper courses are measured with a grade of “C” or higher. 
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body. Finally, many colleges had difficulties involving broad segments of faculty and staff 
in the types of leadership and development opportunities that might ultimately lead to a 
change in practice. 

This chapter describes changes over time in the five key benchmarks listed above for 
the Round 1 colleges in the initiative, providing a preliminary look at the student outcomes at 
these colleges over the course of their five years of planning and implementing Achieving the 
Dream. The analysis suggests that the average trends in the five indicators across the 26 
institutions remained unchanged over time, with a few exceptions. Specifically, the average 
rates for completion of gatekeeper English and coursework completion within two years both 
improved modestly over time. In contrast, the average rates for completion of developmental 
math, developmental English, developmental reading, and gatekeeper math courses remained 
substantially the same throughout the period, as did persistence and completion of a degree or 
certificate. When the data are examined for individual institutions, some show improvements on 
the five indicators. Other institutions, however, show declines on some indicators, and the 
modal pattern is one of no change. 

While these descriptive trends may seem to suggest that Achieving the Dream had few 
positive effects on student outcomes, the results must be interpreted cautiously. Eirst, as noted 
above, many of the strategies directly affecting students remained small in scale during the 
period under study, making it questionable how much they might have affected student out- 
comes during this time. Second, the primary data used in these analyses focus on all entering 
students who were in college for the first time, independent of whether these students were 
targeted by their college to participate in a specific strategy. That is, the analyses use outcomes 
for everyone, not just those targeted by interventions. Given these limitations, it is reasonable to 
expect to see little change in student outcomes at this juncture. 

It is also important to remember that these findings are not causal, but rather simply de- 
scribe movements in student outcomes over time. In order to determine whether the initiative 
caused the patterns that are described above, it is necessary to estimate what would have 
happened in the absence of Achieving the Dream and compare that estimate to what actually 
happened. The best way to determine whether a causal relationship existed between Achieving 
the Dream and changes in student outcomes would be to set up an experimental study that 
randomly assigned colleges to a program group that participated in Achieving the Dream and a 
control group that did not. A random assignment process was not used to select the participating 
colleges, so this method cannot be used for the analysis of the effect of Achieving the Dream on 
student outcomes. However, MDRC did implement a series of random assignment studies of 
particular interventions at several colleges in order to assess how these individual programs 
affected students’ achievement. The results of those studies are summarized in Box 8.1. 
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Box 8.1 



MDRC’s Achieving the Dream impact Studies 

MDRC conducted impact studies at four Round 1 colleges to measure the effects of student 
success strategies on academic performance and persistence outcomes. These studies investi- 
gated the impact of three different strategies on student outcomes: (1) learning communities at 
Hillsborough Community College in Florida and Houston Community College in Texas; (2) a 
college success course at Guilford Technical Community College in North Carolina; and (3) a 
classroom-hased mentoring program at South Texas College. All four impact studies were 
conducted using a random assignment methodology. 

The Beacon Mentoring Program — South Texas College 

In order to address low completion and persistence rates in courses with high enrollment, South 
Texas College administrators established a low-cost, “personal touch” mentoring program. 
Focused on helping students pass math courses that were characterized by high enrollment and 
failure rates. South Texas’s Beacon Mentoring Program assigned trained “Beacon Mentors” 
(volunteer South Texas staff) to a college algebra or developmental math course, in which they 
made brief in-class presentations about student services and academic skills building and 
worked with class instmctors to reach stmgghng or absentee students. Mentors also encour- 
aged students to visit them outside of class for additional academic support. 

The program was evaluated by MDRC during the spring 2008 semester, with over 2,100 stu- 
dents enrolled in 83 math classes participating in the study. Overall, the program did not signif- 
icantly affect students’ academic performances or persistence, though it did increase the num- 
ber of students who accessed tutoring and reduced their withdrawal rates from the math class in 
which they were registered. Some positive academic results were also found among certain 
student subgroups. For instance, part-time students who had a Beacon Mentor were more likely 
to complete their math class and earn more credits, and developmental math students earned 
significantly more credits in their nonmath developmental courses.^ 

Learning Communities — Houston Community College and 
Hillsborough Community College 

With the aim of improving developmental education students’ academic success and en- 
gagement, the learning communities programs at Houston and Hillsborough linked a de- 
velopmental education reading (Hillsborough) or math (Houston) course with a college suc- 
cess course. Both colleges implemented fairly basic learning communities’ models that 
strengthened over the course of the demonstration, as faculty collaborated more frequently to 
integrate the curriculum of the two courses with each other. The student cohorts created by 
co-enrollment were the most salient feature of each program, with students and faculty re- 
porting the development of increased academic and personal support networks.'’ 



(continued) 



123 



Box 8.1 (continued) 



The effect of these learning communities on students’ outcomes varied hy institution. Overall, 
Hillshorough’s program did not have a meaningful impact on students’ academic success (that is, 
there was no difference in student outcomes between those who participated in the program and 
those who did not), though some positive impacts may he seen on the educational outcomes for 
students in the third cohort as faculty collahoration and curricular integration increased.'^ Hou- 
ston’s learning communities, however, had large positive effects on students’ completion of de- 
velopmental math. Students in learning communities also progressed through their developmental 
education course sequence more rapidly."* 

Guilford Technical Community College — “On Course” 

Guilford Tech implemented a one-semester, three-credit student success course based on Skip 
Downing’s On Course™ curriculum, which focuses on fostering students’ academic and socio- 
emotional skills.® The college targeted both new and continuing developmental education students 
to take the course and aimed to improve students’ academic achievement, retention, and campus 
involvement. Three cohorts of students participated in the study from spring 2008 through spring 
2009, and follow-up data are being collected on their academic progress. Results from this study 
will be available in 201 1. 



“Visher, Butcher, and Cema (2010). 

Visher, Schneider, Wathington, and Collado (2010). 
®Weiss, Visher, and Wathington (2010). 

‘'Weissman et al. (forthcoming). 

""Downing (2008). 



The next section outlines the methodology used in this chapter, followed by three sec- 
tions that discuss the trends in student outcomes in detail, examine whether changes in student 
outcomes occur around the start of the initiative, and provide results for two student subgroups. 
The final section summarizes the overall lessons gained through this work. 



Methodology 

Data on the Round 1 colleges obtained from the Achieving the Dream database main- 
tained by JBL Associates are used in the analysis of student indicators.^ The statistics presented 



^^The Achieving the Dream database consists of student-level cohort data that the initiative requires partic- 
ipating colleges to submit annually for six years (Achieving the Dream, 2009). The database consists of 

(continued) 
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in the figures and tables are based on averages of the institutional rates on each measure so that 
colleges with large enrollments are weighted equally to colleges with small enrollments. The 
data consist of average college-level indicators three years before the initiative began (2002 to 
2004, or the “pre-initiative period”) and three years after its implementation (2005 to 2007, or 
the “post-initiative period”)."* Outcomes for each cohort are examined for two years from the 
time students entered college. For example, outcomes for the 2007 cohort are analyzed through 
2009. Since the Round 1 colleges are a diverse group, variation in outcomes are also examined. 
Tests are applied to determine whether the differences in the rates between periods are statisti- 
cally significant,^ and significance levels are adjusted for multiple comparisons.'^ 



Trends in Student Outcomes Among the Round 1 Colleges 

The descriptive analysis of trends looks initially at the rates of indicators over time after 
accounting for possible changes in student composition and differences across states.’ The top 
panel of Figure 8.1 shows the change over time in developmental math completion for students 
requiring remediation. The solid line in the top panel of the figure shows that about 25 percent 
of students in each cohort completed the developmental math sequence within their first two 
years of college. 

While the average trend over time is fairly flat, outcomes at some colleges were much 
higher or lower. The dotted line shows the pattern in developmental course completion over 
time for colleges with student outcomes at the top of the distribution (the ninety-fifth percentile) 
of all Round 1 colleges, while the dashed line shows the outcomes for colleges at the bottom of 
the distribution (the fifth percentile). In 2002, close to 45 percent of students in institutions at 
the top of the distribution completed the developmental math sequence within two years 
compared with 7 percent of students in institutions at the bottom of the distribution. The spread 
between the institutions at the extremes declined somewhat by 2007, with 42 percent of students 
at institutions in the ninety-fifth percentile completing developmental math compared with 
about 6 percent of students at institutions in the bottom fifth percentile. 



student-level outcomes of all degree/certificate-seeking undergraduate students (full and part time) entering an 
institution for the first time either during the fall term or during the 12-month period. 

*The 2007 cohort is used to represent the post-initiative period in most analyses, as that cohort represents 
the fourth year of participation in Achieving the Dream and reflects the latest year for which data are available. 
(See Appendix B for details.) 

^Results that are statistically significant are unlikely to have arisen by chance. 

'’The multiple comparison adjustments follow the guidelines in Schochet (2008). 

’To address the possibility that compositional changes in the student population are likely to occur over 
time, all means are adjusted for gender, age at cohort entry, race/ethnicity, nonresident alien status, and state 
fixed effects. For more detail on the model, see Appendix B. 
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Percentage of students Percentage of students 



Achieving the Dream: Community Colleges Count 



Figure 8.1 

Trends in Developmental Math and Developmental English Completions Among Students 
Referred to Any Developmental English or Math Course, 2002-2007: 

Two-Year Outcomes for Achieving the Dream Round 1 Colleges 
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Figure 8,1 (continued) 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in 
the fall 2002 through fall 2007 cohorts at Achieving the Dream Round 1 colleges. The sohd line shows 
averages adjusted for gender, race/ethnicity, and state fixed effects. To illustrate the variability in these 
outcomes among institutions, the dotted lines show the trend over time among those institutions with student 
outcomes among the highest (the ninety-fifth percentile) of all colleges in Round 1 . The dashed lines show the 
trend among institutions with the lowest student outcomes (the fifth percentile) of all colleges in Round 1 . 

The bottom panel of Figure 8.1 shows the change over time in developmental English 
completion. The solid line shows that average completion rates have remained steady over time 
at around 40 percent. Again, there is substantial variation among institutions, as illustrated by 
institutions at the ninety-fifth and fifth percentiles of the outcomes distribution. The figure 
suggests that there was considerable variation among institutions in the early period and a 
decline in variation toward the end of the period. In fact, the figure suggests a that completion 
rates had converged slightly by 2007, with those institutions at the top of the distribution having 
student completion rates at around 55 percent by the end of the period, while those at the bottom 
of the distribution increased their developmental English completion rates to around 20 percent 
by the end of the period. 

Similar to developmental course completion, trends in “gatekeeper” math — the first 
college-level course in math — show little change over time. The top panel of Eigure 8.2 
reveals that only about 15 percent of students completed the gatekeeper math course within their 
first two years — a very small percentage. This is disconcerting given that many colleges 
require a college-level math course before degree attainment. In contrast, the average gatekee- 
per English completion rate appears to increase by about 5 percentage points over time, as 
shown in the bottom panel of Eigure 8.2. The figure shows that around 30 percent of students in 
the 2002 to 2004 cohorts completed a gatekeeper English course within their first two years; the 
completion rate increased to 34 percent among the 2007 cohort. 

Eigure 8.3 shows changes in course completion over time. The figure shows that approx- 
imately half of students across all cohorts completed 80 percent or more of the courses they 
attempted, although there is some suggestion of a decline in completion among the early cohorts. 
The figure shows that there is a large amount of variation in course completion, with around 86 
percent of students completing at least 80 percent of their attempted courses in institutions at the 
top of the distribution, compared with around 32 percent of students in institutions at the fifth 
percentile of the distribution in 2002. The average rate for completing courses seems to decline 
somewhat among the 2004 cohorts for the institutions at the extremes of the distribution. The rates 
for the later cohorts then appear to stabilize among institutions at the ninety-fifth percentile and 
increase among institutions at the fifth percentile of the distribution. 
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Achieving the Dream: Community Colleges Count 
Figure 8.2 

Trends in Completion of Gatekeeper Math and English Courses Among AU Students, 
2002-2007: Two-Year Outcomes for Achieving the Dream Round 1 Colleges 
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SOURCE: MDRC calculations using Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the 
fall 2002 through faU 2007 cohorts at Achieving the Dream Round 1 colleges. The sohd fine shows averages 
adjusted for gender, race/ethnicity, and state fixed effects. To illustrate the variabihty in these outcomes among 
institutions, the dotted lines show the trend over time among those institutions with student outcomes among the 
highest (the ninety-fifth percentile) of all colleges in Round 1 . The dashed fines show the trend among institutions 
with the lowest student outcomes (the fifth percentile) of all colleges in Round 1 . 
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Achieving the Dream: Community Colleges Count 



Figure 8,3 

Trends in Course Completion, 2002-2007: 

Two-Year Outcomes for Achieving the Dream Round 1 Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: The figure shows the percentage of students completing 80 percent or more of the courses they 
attempted. Calculations for this figure used institutional means for all available data for sample members in the 
fall 2002 through fall 2007 cohorts at Achieving the Dream Round 1 colleges. The solid line shows averages 
adjusted for gender, race/ethnicity, and state fixed effects. To illustrate the variabihty in this outcome among 
institutions, the dotted lines show the trend over time among those institutions with student outcomes among 
the highest (the ninety-fifth percentile) of all colleges in Round 1. The dashed hnes show the trend among 
institutions with the lowest student outcomes (the fifth percentile) of all colleges in Round 1 . 



Overall, this descriptive analysis suggests that the average trend for most indicators is 
fairly stable over time, although there is a considerable amount of variation in rates among the 
Round 1 institutions. This variation is not surprising given the differences among institutions’ 
enrollments, student composition, and student preparation for college, as discussed in Chapter 1. 
Nonetheless, the trend in outcomes among institutions at the extremes of the distribution often 
follows the same pattern of change over time as the average, suggesting that the trend pattern 
was relatively the same across colleges regardless of whether they were at the low or high end 
of the distribution. The next section discusses whether these patterns of change in student 
outcomes are related to participation in the Achieving the Dream initiative. 
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Changes in Trends and Achieving the Dream 

While the analysis above provides useful visuals of change over time, it does not indi- 
cate whether average changes in outcomes between the pre-initiative and post-initiative periods 
are statistically different. This section provides more information about the differences in 
student outcomes during these periods. Box 8.2 explains how to read the tables in this chapter 
that show statistically significant changes.* The first column of Table 8.1 reports adjusted 
averages for the 2002-2004 cohorts (representing the pre-initiative period), and the second 
column reports adjusted averages for the 2007 cohort (representing the post-initiative period). 
The third column reports the difference between the two periods and the fourth column trans- 
lates the percentage point difference into the percentage change over the two periods. 

The first panel of the table shows that developmental math completion rates increased 
by about 4.2 percentage points. The absence of asterisks in the far-right column indicates that 
this difference is not statistically significant, meaning that there is more than a 10 percent 
likelihood that it occurred by chance. The second panel shows the results for gatekeeper math 
and English. As illustrated in Figure 8.2, there is no difference in gatekeeper math completion 
rates between the two periods, but gatekeeper English completion rates increased by 6.4 
percentage points (or 23.7 percent). 

The course completion panel of the table shows that 45.4 percent of students completed 
80 percent or more of their courses within their first two years in college during the pre- 
initiative period and 51.0 percent of students completed most courses in the post-initiative 
period. This finding differs from the slightly declining trend shown in Figure 8.3. The reason for 
the dissimilarity is that the table means are adjusted to show the effect of credit completion after 
accounting for the 2002-2004 (or pre-initiative) pattern in completion rates.'^ After the adjust- 
ment, the table shows that the group of students who completed 80 percent or more of their 
courses in 2007 grew by 5.6 percentage points (for an increase of just over 12 percent). The next 
two panels show that persistence (enrollment from semester to semester) and completion of a 
credential remain largely unchanged. 



*As in the figures, all means in the tables in this chapter are adjusted for gender, age at cohort entry, 
race/ethnicity, nonresident alien status, and state fixed effects. In contrast to the figures, means reported in all 
tables in this chapter are also adjusted for a time trend. Details on the model are provided in Appendix B. 

'^The pre-initiative trend obscures the relationship between student outcomes and the initiative because it 
represents a systematic increase or decrease that was in place prior to the implementation of Achieving the 
Dream. Since the goal of this analysis is to isolate whether changes are related to the implementation of the 
initiative, a time-trend is employed in the model. See Appendix B for details. 
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Box 8.2 



How to Read the Tables in This Chapter 



Most tables in this chapter use a similar format, illustrated below. The abbreviated table shows 
some average student indicators for the 2002-2004 cohorts (which represent the pre-initiative 
period) and the 2007 cohort (which represents the post-initiative period). The first row, for ex- 
ample, shows that 21.0 percent of students in the 2002-2004 cohorts and 25.2 percent of students 
in the 2007 cohort completed developmental math within two years of starting college. 

The “Difference” column in the table shows the differences between the two cohorts’ outcomes 
— that is, the change in outcomes over time. For example, the change in developmental math 
completion can be calculated by subtracting 21.0 percent from 25.2 percent, yielding an increase 
of 4.2 percentage points. The percentage change is derived by dividing the difference by the av- 
erage outcome of the 2002-2004 cohort. For example, 4.2 divided by 21.0 equals 0.201, which 
yields a 20.1 percent increase. 

In the row that displays asterisks, the difference is “statistically significant,” meaning that there is 
only a small probability that it occurred by chance. The number of asterisks indicates the p-value, 
or the likelihood of this chance (that is, the likelihood that the difference is not distinguishable 
from zero). One asterisk corresponds to the 10 percent level, meaning that there is less than a 10 
percent chance that the difference is really zero; two asterisks, less than a 5 percent chance; and 
three asterisks, less than a 1 percent chance. For example, as the third row below shows, the 
change in gatekeeper English completion is 6.4 percentage points, a difference that is statistically 
significant at the 1 percent level, indicating that there is less than a 1 percent chance that the dif- 
ference is really zero. 





Cohort 










Performance Indicators 


2002-2004 


2007 


Difference 


Percentage 

Change 


P- 

Value 


Develoomental courses 












Completion of developmental math 
requirement within two years (%) 


21.0 


25.2 


4.2 


20.1 




Completion of developmental English 
requirement within two years (%) 


37.8 


38.7 


0.9 


2.3 




Completion of gatekeeper 

Enghsh course within first two years (%) 


27.2 


33.6 


6.4 


23.7 


*** 
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Achieving the Dream: Community Colleges Count 
Table 8.1 



Change in Average Two-Year Performance Indicators at Achieving the Dream 
Round 1 Colleges, 2002-2004 Cohorts Compared with 2007 Cohort 



Performance Indicators 


2002-2004 

Cohorts 


2007 

Cohort Difference 


Percentage 

Change 


Sig. 


Referred to anv develonmental eourses“ (%) 


Developmental math 


61.1 


60.9 


-0.2 


-0.3 




Attempted course within first 2 years 


75.8 


70.1 


-5.8 


-7.6 




Completed requirement within first 2 years 


21.0 


25.2 


4.2 


20.1 




Developmental English 


31.7 


35.4 


3.7 


11.7 




Attempted course within first 2 years 


71.3 


64.9 


-6.4 


-9.0 




Completed requirement within first 2 years 


37.8 


38.7 


0.9 


2.3 




Developmental reading 


32.1 


32.8 


0.7 


2.1 




Attempted course within first 2 years 


69.7 


69.8 


0.1 


0.1 




Completed requirement within first 2 years 


37.7 


41.3 


3.5 


9.4 




Successfullv completed aatekeeper courses ^’ '’(%) 


Gatekeeper math within first 2 years 


15.9 


15.5 


-0.4 


-2.4 




Gatekeeper English within first 2 years 


27.2 


33.6 


6.4 


23.7 


*** 


Gatekeeper math within first 2 years, among students 


referred to developmental math‘s 


14.3 


12.2 


-2.1 


-14.5 




Gatekeeper English within first 2 years, among 


students referred to developmental English'^ 


25.1 


28.2 


3.1 


12.4 




Course completion'* 


Attempted credits within first 2 years 


30.0 


31.0 


1.0 


3.2 




Completed credits within first 2 years 


22.0 


24.5 


2.5 


11.5 


** 


Ratio of completed credits to attempted credits within 


first 2 years (%) 


66.2 


72.4 


6.1 


9.2 


** 


Completed 80 percent or more of attempted credits 


within first 2 years (%) 


45.4 


51.0 


5.6 


12.3 


** 


Achieved 20 credit hour milestone within 2 years (%) 


47.4 


55.6 


8.2 


17.3 


** 


Persistence (%) 


Enrolled in fall of second year 


46.2 


49.5 


3.3 


7.2 




Enrolled in at least one semester in second year 


51.4 


54.2 


2.8 


5.4 




Completions (%) 


Completed a credential by the end of the second year 


5.2 


7.6 


2.4 


45.4 




Completed an associate's degree 


3.2 


4.1 


1.0 


31.0 




Completed a certificate or diploma 


2.1 


3.5 


1.4 


64.7 





(continued) 
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Table 8.1 (continued) 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for this table used institutional means for all available data for sample members in the 
fall 2002 through fall 2007 cohorts at the Achieving the Dream Round 1 colleges. While Alamo Community 
College District is considered one site for the Achieving the Dream initiative, it has 4 colleges that report data 
separately to the database, bringing the total number of data-reporting institutions from 26 to 29. Because 2 of 
these 29 colleges (Central New Mexico Community College and New Mexico State University-Dona Ana) did 
not submit 2008 data to JBL, the total sample size for this table is 27 colleges. 

Data for the fall 2002 cohort at Guilford Technical Community College were not submitted. 

Entries show estimated difference in probabilities from a linear regression that controls for gender, 
race/ethnicity, a time trend, and state fixed effects. Statistical significance levels, shown in the Sig. column, are 
indicated as: *** = 1 percent; ** = 5 percent; and * = 10 percent. 

To control for spurious findings resulting from multiple comparisons, significance levels for individual 
variables are shown only if the aggregate domain measure was statistically significant with a p-value below the 
Benjamini-Hochberg maximum (Schochet, 2008). 

Standard errors are clustered by institution. 

“Grade of "C" or better must be earned to successfully complete a course. 

'’Gatekeeper math is the first college-level math course at the college; gatekeeper English is the first 
college-level English course at the college. 

“Calculations used data for sample members who have developmental referral data. 

“'Completion with a grade of "D" or better is reported for this block of outcomes. Credits completed with a 
grade of "C" or better was not introduced as a variable until the 2007 JBL submission guide was released. 



Variation in Differences Over Time 

Although differences for most outcomes are small (and positive) or show no change, 
the averages could mask variation across the institutions. Figure 8.4 examines variation across 
Round 1 institutions in the change in gatekeeper English completion rates over the two periods, 
and shows a moderate amount of variation across institutions. For example, the largest increase 
in gatekeeper English completion was 18.6 percentage points (a 65 percent increase for this 
institution) and the largest decrease in gatekeeper English completion was 14.6 percentage 
points (a 32.5 percent decrease for this institution). Of the 27 institutions examined, 1 1 of the 
institutional-level differences were statistically significant (indicated by the black bars in 
the figure). 

There is less variation in the distribution of differences over time in fall-to-fall persis- 
tence, as shown in Figure 8.5. The largest (statistically significant) increase in this measure of 
persistence was 10.3 percentage points (representing a 22 percent increase at this institution), 
and the largest (statistically significant) decrease in this measure was 12.0 percentage points 



*®While 26 institutions participated in Achieving the Dream, Alamo Community College District has four 
colleges that report data separately to the database, bringing the total number of data-reporting institutions from 
26 to 29. Because two of these 29 colleges (Central New Mexico Community College and New Mexico State- 
Dona Ana) did not submit 2008 data to JBL, the total number of institutions in this analysis is 27 colleges. 
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Achieving the Dream: Community Colleges Count 
Figure 8,4 

Distribution of Pre-Initiative and Post-Initiative Differences in Completion of 
Gatekeeper English at Achieving the Dream Round I Colleges, 2002-2007 




SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: The figure shows the percentage point difference in the completion of gatekeeper English within two 
years between the averages for the 2002-2004 cohorts and the 2007 cohort. The Benjamini-Hochberg correc- 
tion for multiple hypothesis testing was applied. The black bars indicate differences that are statistically 
significant at the corrected p-value or lower (Schochet, 2008).The white bar shows the overall average 
difference of 6.4 percentage points presented in Table 8.1, which is statistically significant at the 1 percent 
level. Calculations for this figure used institutional means for all available data for sample members in the fall 
2002 through fall 2007 cohorts at Achieving the Dream Round 1 colleges. The averages that generated the 
differences were adjusted for gender, race/ethnicity, state fixed effects, and a linear time trend. 



(representing a 28 percent decrease at this institution). The results of an analysis of variation in 
other outcomes are shown in Figure 8.6. Most outcomes vary substantially across institutions, 
with most institutions falhng into the category of having no change in student outcomes over time. 
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Achieving the Dream: Community Colleges Count 
Figure 8.5 

Distribution of Pre-Initiative and Post-Initiative Differences in Fall-to-Fall Persistence 
at Achieving the Dream Round 1 Colleges, 2002-2007 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: The figure shows the percentage point difference between the averages for the 2002-2004 cohorts and 
the 2007 cohort in fall-to-fall persistence. The Benjamini-Hochberg correction for multiple hypothesis testing 
was applied. The black bars indicate differences that are statistically significant at the corrected p-value or 
lower (Schochet, 2008). The white bar shows the overall average difference of 3.3 percentage points presented 
in Table 8.1. Calculations for this figure used institutional means for all available data for sample members in 
the fall 2002 through fall 2007 cohorts at Achieving the Dream Round 1 colleges. The averages that generated 
the differences were adjusted for gender, race/ethnicity, state fixed effects, and a linear time trend. 



In addition to the foregoing analysis, an exploratory analysis was conducted to examine 
whether the student outcomes at colleges with more dramatic changes in their culture of evidence 
may have differed from the overall group. As discussed in Chapter 7, field research of the 
initiative revealed some unevenness in the Round 1 colleges’ adoption of practices associated 
with a culture of evidence. Of all of the colleges examined, four institutions were found to have 
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Achieving the Dream: Community Colleges Count 
Figure 8,6 

Distribution of Institution-Level Changes in Two-Year, Interim Performance Indicators 
at Achieving the Dream Round 1 Colleges, 2002-2004 Compared with 2007 



s 

Z 




Math English Reading Math English 80% or more Eall-fall Any 

credential 

SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 



NOTES: Calculations for this figure used institutional means for all available data for sample members in 
the fall 2002 through fall 2007 cohorts at Achieving the Dream Round 1 colleges. 

Some colleges did not report into the database on some measures. 

Bars show the number of differences that are positive and statistically significant ("Positive Change"), 
not significant ("No Change"), or negative and statistically significant ("Negative Change"). The 
Benjamini-Hochberg correction for multiple hypothesis testing was applied. Statistical significance is 
characterized as p-values below the corrected maximum (Schochet, 2008). 

Eor the measures of "developmental requirement completion" and "gatekeeper course completion," a 
grade of "C" or better must be earned to successfully complete a course. Gatekeeper math is the first 
college-level math course at the college; gatekeeper English is the first college -level English course. 

Eor the "80% or more course completion" measure, completion with a grade of "D" or better is reported. 
Credits completed with a grade of "C" or better was not introduced as a variable until the 2007 JBL submis- 
sion guide was released. 
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a weak culture of evidence one year after the initiative began, but a strong culture of evidence 
three years later. These “star” institutions thus showed the greatest level of progress during their 
time in Achieving the Dream and, in interviews and focus groups, attributed many of their 
changes to their work in Achieving the Dream. Given their high level of progress, these colleges 
might be expected to have different student outcome trends from other Round 1 colleges. Howev- 
er, the results of the exploratory analyses of these star colleges suggest that the distribution of 
outcomes at these colleges is no different from those of other Round 1 institutions." That is, these 
continuously improving colleges also experienced increases, decreases, and lack of change across 
the various indicators of student outcomes. This finding suggests that the trends in student 
outcomes did not vary based on improvements to the practices associated with a strong culture of 
evidence, at least not during the time period covered in this report. 

Variation in Differences by Student Subgroups 

As noted in previous chapters, improving student outcomes and closing the gap in 
achievement by income and race or ethnicity are major goals of the initiative. As such, this 
section explores whether the income, racial, and ethnic gaps in achievement that are apparent 
early in the initiative decreased during the colleges’ participation in Achieving the Dream. 

Differences by Pell Grant Receipt Status 

Numerous studies have shown that low-income students enroll in college at lower rates, 
are more likely to “stop out” (that is, intermpt their studies) or drop out of college, and take 
significantly longer to complete a degree or certificate if they do remain in school.'^ Therefore, 
improving outcomes among this group may have lasting effects. Table 8.2 shows changes in 
student outcomes over time by Pell Grant receipt status, a proxy for low-income status. The first 
set of three columns reports outcomes for students who did not receive a Pell Grant. The differ- 
ences shown in the third column reveal no statistically significant change over time in the Achiev- 
ing the Dream indicators — that is, from the pre-initiative period (cohorts 2002-2004) to the post- 
initiative period (cohort 2007). 

The next set of three columns shows the changes in student outcomes for Pell Grant reci- 
pients. When comparing Pell Grant recipients with those who did not receive a Pell Grant in the 
pre-initiative period, on average, the grant recipients are referred more often to developmental 
coursework and are less likely to complete gatekeeper math and the credits they attempted within 
their first two years, but are more likely to complete gatekeeper English within their first two 
years. These patterns remained largely the same in the post-initiative period, with some 



"These analyses are not shown in this report. 

'^For example, see Berker, He, and Cataldi (2002); Brock et al. (2007). 
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Achieving the Dream: Community Colleges Count 
Table 8.2 

Change in Average Two-Year Performance Indicators 
at Achieving the Dream Round 1 Colleges, by Pell Grant Status, 
2002-2004 Cohorts Compared with 2007 Cohort 



No Pell Grant Receipt Pell Grant Receipt 





2002-2004 


2007 




2002-2004 


2007 




Performance Indicators 


Cohorts 


Cohort 


Diff. 


Cohorts 


Cohort 


Diff 


Referred to anv develonmental courses^ (%) 

Developmental math 


51.5 


57.7 


6.2 


71.2 


67.2 


-4.0 


Attempted course within first 2 years 


68.4 


64.0 


-4.4 


80.7 


77.7 


-3.0 


Completed requirement within first 2 years 


19.2 


23.8 


4.6 


21.9 


27.3 


5.3 


Developmental English 


24.4 


33.9 


9.5 


39.8 


41.0 


1.2 


Attempted course within first 2 years 


62.9 


62.9 


0.0 


76.2 


71.8 


-4.4 


Completed requirement within first 2 years 


36.6 


31.1 


-5.5 


41.3 


43.9 


2.5 


Developmental reading 


23.2 


28.4 


5.2 


41.1 


38.6 


-2.5 


Attempted course within first 2 years 


61.4 


64.8 


3.4 


73.9 


75.4 


1.5 


Completed requirement within first 2 years 


33.5 


36.7 


3.2 


40.4 


46.7 


6.3 


Successfullv comnleted 2atekeener 
courses^’’'’ (%) 

Gatekeeper math within first 2 years 


18.4 


16.3 


-2.1 


16.5 


15.2 


-1.3 


Gatekeeper English within first 2 years 
Gatekeeper math within first 2 years, among 


28.7 


29.4 


0.8 


30.7 


37.4 


6.8 


students referred to developmental mattf 
Gatekeeper English within first 2 years, 


18.6 


8.7 


-9.9 


14.2 


11.7 


-2.5 


among students referred to developmental 
English' 


30.8 


19.8 


-10.9 


24.7 


34.4 


9.7 


Course completion'' 

Attempted credits within first 2 years 


28.4 


27.6 


-0.8 


33.9 


35.5 


1.6 


Completed credits within first 2 years 
Ratio of completed credits to attempted 


22.2 


21.8 


-0.4 


24.3 


27.2 


2.9 


credits within first 2 years (%) 
Completed 80 percent or more of 


71.7 


70.9 


-0.8 


64.7 


69.8 


5.1 


attempted credits within first 2 years (%) 
Achieved 20 credit hour milestone within 


56.3 


58.2 


1.9 


43.1 


49.6 


6.5 


2 years (%) 


43.7 


44.7 


1.0 


52.5 


58.2 


5.7 


Persistence (%) 

Enrolled in fall of second year 
Enrolled in at least one semester in second 


47.1 


42.4 


-4.6 


52.1 


55.3 


3.1 


year 


51.8 


47.0 


-4.8 


57.6 


61.7 


4.1 


Comnletions (%) 

Completed a credential by the end of the 
second year 


7.3 


6.2 


-1.1 


4.1 


7.6 


3.5 


Completed an associate's degree 


5.0 


2.9 


-2.1 


2.3 


4.1 


1.8 


Completed a certificate or diploma 


2.3 


3.4 


1.1 


1.8 


3.6 


1.8 



(continued) 
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Table 8.2 (continued) 

SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for this table used institutional means for all available data for sample members in the fall 
2002 through fall 2007 cohorts at the Achieving the Dream Round 1 colleges. While Alamo Community College 
District is considered one site for the Achieving the Dream initiative, it has 4 colleges that report data separately to 
the database, bringing the total number of data-reporting institutions from 26 to 29. Because 2 of these 29 colleges 
(Central New Mexico Community College and New Mexico State University-Dona Ana) did not submit 2008 data 
to JBL, the total sample size for this table is 27 colleges. 

Data for the fall 2002 cohort at Guilford Technical Community College were not submitted. 

Entries show estimated difference in probabilities from a linear regression that controls for gender, 
race/ethnicity, a time trend, and state fixed effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; and * = 10 percent. 

Differences in indicators between Pell recipients and other students were tested for statistical significance; no 
statistically significant differences were found. 

To control for spurious findings resulting from multiple comparisons, significance levels for individual 
variables are shown only if the aggregate domain measure was statistically significant with a p-value below the 
Benjamini-Hochberg maximum (Schochet, 2008). 

Standard errors are clustered by institution. 

“Grade of "C" or better must be earned to successfully complete a course. 

'’Gatekeeper math is the first college-level math course at the college; gatekeeper English is the first college- 
level English course at the college. 

“Calculations used data for sample members who have developmental referral data. 

“'Completion with a grade of "D" or better is reported for this block of outcomes. Credits completed with a 
grade of "C" or better was not introduced as a variable until the 2007 JBL submission guide was released. 

improvements in average indicators for Pell Grant recipients in the post-initiative period. Howev- 
er, when comparing the differences across these two groups (not shown), the changes were not 
large enough to result in statistically significant reductions in the achievement gap between 
students who received a Pell Grant and those who did not.'^ 

Differences by Race and Ethnicity 

The first set of columns in Table 8.3 reports outcomes for white students, who make up 
the plurality of the student body at Round 1 colleges, on average. The table shows that white 
students experienced statistically significant improvement over time in successfully completing 
gatekeeper English and completing credits attempted. The next three columns show outcomes for 
black students, which reveal significant increases in course completion. The last set of columns 
shows outcomes for Hispanic students (who may be of any race). Unlike the previous groups, the 
indicators show no significant change over time. These findings suggest that there were some 
improvements in white and black students’ course completions, but when looked at on the 
whole, there were few improvements across most of these indicators for these different student 
subgroups. Additionally, tests comparing the changes in white, black, and Hispanic students’ 



^The p-values from the formal tests of the differences between the two groups exceed conventional levels 
(that is, the p value is greater than 0.10), indicating that the differences could have occurred by chance. 



139 




Achieving the Dream: Community Colleges Count 



M 

"o 



c 

o t; 

Pi o 

s ■§ 

^ rj 
u 

fi ® 
o 
a 

.s ^ 

>► 



00 

s 

H 



JS 

u 



© 

fl 

© 

u 

fl 



£ 

u 

Ph 

ct 

<u 



o 

H 



U 



o> 

es 

a 

S 

o 

U 



o 

o 

U 

o 

o 

rji 

r<i 

o 

o 



■3 

JS 

© 
© 
C9 

© 

OX 

2^ 

© 

?► 

<! 



© 

ox 

G 

G 



0 


3 : 


04 


0 




u 




c/3 


0 




0 


0 


04 


J= 


1 

04 

0 


0 

u 


0 




04 





^ t: 

S ® 

P 

CN O 

u 

^ i 2 
o t^ 
o o 

O'! 

sa 

o 

CN 



p 


CO 


CO 






wo 


p 


p 


p 


CO 


d 


d 


NO 


CO 


d 


H 


CO 


CO 








' 


' 


' 


p 






00 


p 


p 


4> 




p 


CO 




CO 


oi 


wS 


.-H 


On 


oi 


CO 


On 






NO 


i> 


CO 


04 


NO 


CO 


04 


O' 




00 


p 


0 




O' 


in 


On 






ON 


3- 


d 


NO 


d 


CO 


d 


0 


WO 


in 


I> 


04 


CO 


O' 


CO 




NO 


CO 



I> p 



i> p 

»n i/S ^ 

IT) O — H 



<N 10 
^ G in 
00 ^ 



p 

ON I> 



p ^ to 
oi CN NO 
m NO 






5 

0 
cj 

13 

fl 

0 ) 

1 

0 ) 

> 

0 ) 

-© 

CQ 

O 



o 

^ rv| 

I » 

O Ui 
>v 



CN 



c 

, ‘I 

1 O 



c 

■l 

c 

(U 

s 

(U 

•in 

'3 

cr 

(D 



o 
o 
o 
T3 

0 

1 I 

< c3 



O) rn 

<n r4 



p p 

<ri r4 

m NO <T) 



■p «n cs p ^ cs 

oi ON rj wS wS 

NO c<N tj- <D 



W-H 


f-H 


p 


p 


p 


00 


P 


p 


CO 


5 


d 


oi 


wo 


CO 


wo 


H-; 




p 


u 


p 


p 


p 


p 


NO 




On 


CO 


0 


CO 


0 ^ 


CO 




d 


On 


d 


CO 


0 

u 


wo 


NO 


04 


04 


NO 


CO 


04 


NO 


c/3 


00 


wo 








p 


On 


in 




CO 


d 


oi 


d 


NO 


0 


On 




J= 

0 

U 


•o 


o- 


04 


04 


NO 


CO 




NO 



p 

c4 



a> 

1 “ 
>'« 

c 

2 ^ 

S-H ■4— > 

1'! 

^11 
22 ^ c 
o 
bO 

c 23 D 
PQ 3 cr 
-^00 
cd o 

g ^ I 

a 

^ 1 s 

(D ti o 

Q 



toJD • 

.s 

cd 

dJ 



tT CN 



3 

o 

^ CN 
5-h 

2 e 

>'C 

c 

e S 

«'! 
^ C 

D 



3 

cr 

o 



g ^ a 

a 

S" a a^ 
^ « a 

o ti o 

s C U 

Q 






I 



p 

CN 



CO cN m 

O NO ^ 



p ■^. 

06 o 

^ CO 



iri 06 



CN 

d 



p p 
CO i> 

^ CN 



NO 00 
ON 



wo 
d ON 



^ p 
06 NO 



-o 

3 



W) 

c 

o 

s 

e « 

1 1 „ 
=2 (N a 
c }P. 



3 
^ 3 



•S .5 B 

d ^ -5 & 

5 ;S 5 •§ 

■S "Ob'S u 

3 c ^ r3 

a w a o 

Vh U }-i 

dJ O 0 -rt 
Oh Qh Oh 



o <0 
o <0 
3 3 

o a 



NO 






04 

04 



in 



o 

S 'bb 

c 

c« PJ 

o _5 

^ c 

<N ^ 

23 ^ 
- <u 

.s > 

.tn ^ 

^ B 

'O 

.23 o 

u 

a P 
PJ p 



(D cci 

3 
<U 

-o 
3 



o 

o 

3 

a 



p 00 
NO CO 



<N 00 O 
On 
NO 



NO ON 
<N 



CO 

d 04 



00 ON 
06 CO 



p P 
oi CO 
CO 04 



00 

oi NO 



p Tt 

00 oi 

04 04 



in NO 
04 ^ 



P 

oi 



r-l in 
CO 04 



in p 

ON oi 

04 04 



’O 

dJ 



’O 

(U 



^ & 

s 

>v dJ 



04 



.S T3 

C4 ^ 



I ^ Cfl 

I ^ 

1 dJ 
' .ts Vh 

“ a 

'3 ^ 

a & 

“ a 

-TO O 

(U o 

. “ ‘S 

^■S 

O k! 

U CK 



140 



within first 2 years (%) 70.5 75.9 5.4 ** 56.3 72.3 16.0 *** 66.8 

Completed 80 percent or more of attempted credits 

within first 2 years (%) 54.3 61.5 7.2** 38.4 50.1 11.7 * 49.0 

Achieved 20 credit hour milestone within 2 years (%) 46.7 52.6 6.0 *** 32.0 49.4 17.4 ** 46.6 




White, non-Hispanic Black, non-Hispanic Hispanic 

2002-2004 2W7 2002-2004 2007 2002-2004 2007 

Performance Indicators Cohorts Cohort Diff. Cohorts Cohort Diff. Cohorts Cohort Diff. 
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‘‘Completion with a grade of "D" or better is reported for this block of outcomes. Credits completed with a grade of "C" or better was not introduced as a 
variable until the 2007 JBL submission guide was released. 




outcomes (not shown) revealed no significant differences, and as a result, the achievement gap 
remained largely unchanged. 

Is There an Association Between Changes in Trends and Achieving the 
Dream? 

While the descriptive analyses in the figures and Tables 8.1, 8.2, and 8.3 seem to sug- 
gest that Achieving the Dream has had positive effects on some student outcomes and no effect 
on others, it is important to remember that these findings are not causal. That is, since trends in 
student outcomes may change independently of Achieving the Dream, simple changes in 
student outcome trends cannot be interpreted as indicating that the initiative had a positive or 
negative effect on student outcomes. First, the presented measures do not account for other 
processes that could have been implemented around the same time as Achieving the Dream and 
that could have affected student outcomes in the absence of the initiative. Second, these 
indicators may gradually improve over time as a result of natural processes. For example, a 
higher proportion of students will pass developmental math in two years than in one year simply 
because of more time elapsing.'^ 

Finally, another reason to avoid drawing a causal inference from these findings is that 
the participating Achieving the Dream schools were chosen through a selective process that 
automatically distinguishes them as being different from schools that were not selected or did 
not apply. Therefore, differences in outcomes at the Achieving the Dream schools may have 
been lower, the same, or higher independent of the implementation of Achieving the Dream. In 
order to derive the causal effect of Achieving the Dream, it is necessary to compare the actual 
outcomes from Achieving the Dream schools with outcomes that represent what would have 
happened in the absence of the initiative. 



Summary 

While Achieving the Dream had many successes with the Round 1 colleges, the colleg- 
es varied in their ability to adopt all facets of the five-step model. For instance, many of the 



'“'For example, the percentage of students who are referred to the lowest levels of developmental math 
could have declined steadily if the students entering the colleges each year had higher levels of math proficien- 
cy. This trend could result in a larger number of students completing the developmental math sequence within 
the first year. Given that such alternative explanations cannot be ruled out, simple differences in outcomes 
should be regarded only as a descriptive measure of change over time and not a result of the Achieving the 
Dream initiative. 

'^These are examples of two particular threats to the internal vahdity of the analysis. Internal vahdity re- 
fers to the extent to which the independent variable can be proven to have produced the observed effect. The 
particular threats of history and maturation that are described in the text are common problems in before-and- 
after comparisons such as the ones presented in this chapter (Cook and Campbell, 1979). 
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institutions were still facing important challenges in scaling up reforms that might directly affect 
students’ achievement as late as 2008-2009, or the colleges’ fourth year of implementation (the 
end point for this analysis of student outcomes). In addition, many colleges struggled to under- 
take certain forms of data analysis, such as strategy evaluations, making it difficult to know 
whether and how their strategies might be influencing students’ success. As a result of these and 
other factors, the majority of colleges’ strategies also tended to remain small in scale, affecting 
only selected pockets of the student body. 

Given these issues, much variation existed in the student outcomes across these colleg- 
es, and few changes were seen overall in Achieving the Dream’s five indicators of student 
success. Research into organizational change, specifically change that is associated with the 
delivery of education, has shown repeatedly that time is needed for the organizational restruc- 
turing to be manifested in changes in student outcomes. That research, taken together with the 
unevenness in colleges’ implementation of Achieving the Dream’s principles and practices, 
suggests that more time may be needed to see real changes in students’ achievement. 

However, some of these findings also imply that Achieving the Dream may need to modi- 
fy some of its practices to see more substantial changes in student outcomes. For instance, though 
closing achievement gaps was a stated purpose of the initiative, relatively few changes were seen 
in achievement gaps among low-income students and students of color. Similarly, the analysis of 
those colleges that made the most progress in building a culture of evidence suggests that changes 
to colleges’ institutional practices alone may not be enough to influence students’ outcomes. Such 
findings reveal that Achieving the Dream may need to focus greater attention on helping colleges 
develop more programs and practices that affect students’ achievement directly and that aim to 
reduce the differences in outcomes among students by race, ethnicity, and income level. 

Despite these challenges, the positive movement in students’ completion of gatekeeper 
English courses and overall course completion does suggest some room for cautious optimism. 
Field research at the Round 1 colleges indicated that many colleges had made a substantial shift in 
how they addressed students’ success, even if their efforts had not yet translated into sizeable 
programs and interventions for students. As a result, colleges may be better poised in the future to 
develop a more limited set of interventions and focus on scaling up those programs to reach larger 
numbers of students. Similarly, colleges’ increased data capacity may better position them to 
evaluate the success of such strategies and make modifications to ensure they are resulting in 
improved student achievement. Perhaps this stronger focus on large-scale student interventions, 
along with a continued strengthening of the culture of evidence at these colleges, will produce 
more substantial shifts in student outcomes in the future. 



'®Quint (2006); Payzant (2007). 
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Chapter 9 



Instituting Large-Scale Reform: 
Achieving the Dream in Context 



As observed throughout this report, many of the Round 1 colleges in Achieving the 
Dream made substantial progress in developing more evidence-based systems that are aimed at 
improving students’ success rates. Over 75 percent of the Round 1 colleges improved their 
culture of evidence over the course of their five-year participation in the initiative, and only a 
handful of colleges made little progress in implementing the initiative’s model for institutional 
improvement. Indeed, a number of colleges had great success with the initiative, with over 40 
percent of colleges instituting nearly all of the practices associated with a strong culture of 
evidence and many attributing their improvements to Achieving the Dream. Overall, most of the 
Round 1 colleges made a number of enhancements to their institutions, including more sophisti- 
cated methods for data analysis, numerous interventions aimed at increasing students’ achieve- 
ment, and more efficient systems for monitoring those efforts. 

Lumina Foundation for Education, Achieving the Dream, and the colleges themselves 
took bold steps to try to improve student achievement, and they have met with a number of 
successes. While community colleges are a prominent part of the national conversation today 
about how to improve the academic achievement of low-income populations in particular, few 
organizations were concentrating on these institutions before 2004, and even fewer were 
devoting substantial funds to their improvement. In the short time since the initiative began, 
Lumina, Achieving the Dream, and the participating colleges made substantial strides in 
bringing community colleges to the forefront. 

While Achieving the Dream has made significant progress with community colleges, 
the initiative, like its participating institutions, also has room for improvement. As reported here, 
a number of challenges still exist for the community colleges that are participating in Achieving 
the Dream. For instance, outside of implementing colleges’ strategies, the faculty and staff at 
many of the Round 1 colleges tended to be less involved than administrators in the colleges’ 
overall improvement work. Similarly, only a handful of colleges attempted to engage their part- 
time faculty and staff, who most often teach the developmental and gatekeeper courses that are 
so important to students’ ultimate success in community colleges. Additionally, while most 
colleges had expanded at least one strategy, the majority of strategies at these schools remained 
small in scale, leaving large proportions of students relatively untouched by the colleges’ 
Achieving the Dream work. Finally, Achieving the Dream colleges have seen relatively few 
changes in their student outcomes over the course of their five years in the initiative, raising a 
number of questions as to how institutions’ reforms are affecting students on the ground. 
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These issues represent important challenges to Achieving the Dream’s vision for im- 
proved student achievement; however, they should also be placed in the context of the overall 
struggles facing the U.S. education system as a whole. Elementary and secondary schools have 
long faced an uphill battle in their efforts to improve students’ achievement, with only a relative 
few achieving large-scale reforms that successfully increased students’ academic skills or 
graduation rates.' Far more have been mired in many of the same struggles that Achieving the 
Dream colleges face, including students with major academic challenges, poor funding streams, 
difficulties bringing programs to scale, and a lack of faculty or staff engagement in larger school 
reform efforts.^ Additionally, school systems working with adolescent populations, such as 
middle and high schools, tend to have the greatest struggle with improving student achievement, 
revealing the key challenge facing community colleges.^ 

Achieving the Dream’s four principles and 34 criteria for increasing student success 
through institutional improvement have provided a broad stmcture for dealing with many of 
these challenges. Similarly, the initiative’s five-step process for increasing students’ achieve- 
ment, which focuses on honing colleges’ leadership commitment, data analysis, engagement of 
internal and external stakeholders, implementation of interventions, and institutionalization of 
successful practices, has offered a strong model for this change process. However, lessons from 
the institutional change and K-12 literature on whole systems reform also provide useful 
insights into ways in which the Achieving the Dream model might be expanded to have an even 
greater impact on students’ achievement. Drawing from that literature, as well as from the 
interim findings from the Round 1 colleges presented in this report, this final chapter provides 
several recommendations for improving Achieving the Dream’s model for institutional change. 



Improvements to the Leadership Model in Achieving the Dream 

• Broaden the leadership model to include faculty, staff, and lower-level 
administrators. 

The model for institutional improvement under Achieving the Dream has focused 
heavily on the commitment and involvement of presidents and senior administrators in 
colleges’ efforts to drive changes. Leadership commitment makes up the first step of the 
initiative’s five-step model for increasing students’ success and is the first principle in 
Achieving the Dream’s recommended practices for institutional improvement. The initiative 



'Kemple, Herlihy, and Smith (2005); Snipes, Doolittle, and Herlihy (2002); Bloom, Thompson, and Un- 
terman (2010). 

^Berends, Kirby, Naftel, and McKelvey (2001); Elmore and Burney (1996); Elmore (2000); Quint (2006). 
^Snipes and Casserly (2004); Quint (2006); Balfanz and Legters (2005). 
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has also provided key supports to enhance senior leaders’ involvement, including a presi- 
dential coach and a number of annual events, such as Trustee Institutes and focused discus- 
sions at the annual Strategy Institutes. 

While presidential involvement further enhanced a number of colleges’ work in 
Achieving the Dream, the involvement of senior leaders, mid- and lower-level administrators, 
faculty, and staff in the leadership process was often a stronger determinant of a college’s 
success in achieving broad-based change. For instance, many colleges that built up their data- 
based decision-making capacity had senior, or even lower-level, administrators leading their 
Achieving the Dream work, with presidents who were only minimally involved. Alternately, 
several colleges that struggled with Achieving the Dream had presidents who were leading the 
colleges’ work. Finally, while many strong Achieving the Dream colleges had visionary 
presidents, a number of colleges with less influential leaders also made great progress in 
building a culture of evidence. Therefore, while presidential leadership may be useful in 
colleges’ efforts to build a culture of evidence, the regular involvement of the college president 
does not appear to be the defining factor of colleges’ success in fostering institutional change. 

In contrast, a number of K-12 reform efforts have tended to focus on reforms that 
change the on-the-ground interactions among faculty, staff, and students, primarily through a 
revision of classroom practices and curriculum."^ Though improving students’ academic out- 
comes remains difficult for many K-12 reforms, schools that have seen some successes with 
increasing students’ achievement or workforce outcomes — such as New York City’s “small 
schools of choice,” Talent Development High Schools, and Career Academies — engaged 
teachers and other front-line staff as the key players in students’ educational lives. ^ These 
initiatives have tended to focus on the classroom as the primary vehicle for improving students’ 
success, with a heavy focus on professional development and other leadership models that will 
directly affect instructors’ practice and students’ learning. 

"^Bryk et al. (2009); Stigler and Hiebert (1999); Elmore (2000); Quint, Thompson, and Bald (2008); Corrin 
et al. (2008); Bryk and Schneider (2002). 

^New York City’s “small schools of choice” are small, academically nonselective, four-year public high 
schools for students in grades 9 through 12, open to students at all levels of academic achievement and located 
in historically disadvantaged communities. See Bloom, Thompson, and Unterman (2010). 

Talent Development, which targets some of the most troubled schools in the country, seeks to raise the 
expectations of teachers and students, with the ultimate goal of preparing all students for postsecondary 
education and employment. See Kemple, Herhhy, and Smith (2005). 

The Career Academy approach is a widely established high school reform initiative aimed at improv- 
ing students’ performance in high school and providing them with clearer pathways to postsecondary 
education and careers. Career Academies are typically organized as schools-within-schools in an effort to 
create more supportive teaching and learning communities; they have a career theme and attempt to 
integrate a college preparatory academic curriculum with more applied, occupation-related courses; and 
they establish partnerships with local employers as a means of increasing students’ awareness. See 
Kemple (2008); Kemple and Rock (1996); Quint, Bloom, Black, and Stephens (2005). 
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While Achieving the Dream has promoted the inclusion of faculty and staff in colleges’ 
core teams and developed several principles focused on professional development, the initiative 
to date has provided fewer formalized systems for integrating faculty and staff into the work to 
bring about change, instead leaving their inclusion largely up to individual college administra- 
tors, with variable results. Given the primary role that faculty and staff play in teaching and 
supporting students’ learning, the initiative should focus more attention on directly engaging 
these personnel as leaders in the colleges’ reform process. For instance, the initiative might 
involve mid-level managers more directly, such as department heads and deans, as a way to 
better affect faculty and staff practice. College faculty and staff could work with coaches to help 
amplify their voices in the institutional improvement process. Similarly, the initiative might 
develop more systematic ways for faculty and staff to share and build upon their knowledge of 
promising practices for improving student success and employ those individuals as mentors to 
new Achieving the Dream colleges. Finally, the initiative might designate a leadership team of 
faculty and staff that would work in tandem with senior administrators in developing the 
colleges’ improvement agenda. Their valuable knowledge of students’ successes and challenges 
would help bring a critical perspective to colleges’ efforts to improve students’ achievement. 

• Focus more directly on improving instruction and student services. 

To date. Achieving the Dream has focused on helping colleges improve student 
achievement by encouraging faculty, staff, and administrators to analyze data on student 
outcomes to identify priorities for reform and develop interventions to improve students’ 
achievement. As noted in the preceding chapters, these recommendations have succeeded at 
increasing college personnel’s awareness of their students’ poor outcomes; however, they have 
less often led to the colleges’ understanding of what practices they might use to improve these 
students’ achievement. For instance, colleges often used their data analysis to identify broad 
areas of focus, such as developmental education, but then implemented a smattering of strate- 
gies that may or may not have addressed the core challenges these students were facing. While 
the initiative encourages colleges to use research and data to identify successful practices, in 
actuality many Round 1 colleges chose instead to implement strategies that were popular at 
other schools or that they had heard about at the initiative’s Strategy Institutes. Additionally, 
few of those interventions focused on changing the instruction in content courses; instead, most 
concentrated on offering supplemental supports to students, such as tutoring, additional advis- 
ing, or student success courses. 

This disconnect between colleges’ data analysis and choice of interventions highlights a 
chasm in Achieving the Dream’s theory of change, which provides clear methods for colleges 
to identify their problems but offers less guidance on what practices might help resolve those 
problems. Recent research on K-12 reforms suggests that data-based decision making may not 
be enough to improve student achievement and, instead, schools should focus more sharply on 
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instruction and the strategies for overcoming students’ learning difficulties.*’ Key to such efforts 
is focusing on a revision of the classroom practices and instmction that are most likely to 
change students’ academic outcomes. For instance, in efforts to improve students’ learning, the 
First Things First initiative — a comprehensive school reform model that seeks to address the 
impersonal nature and poor performance of many secondary schools serving disadvantaged 
students — included the implementation of professional development activities that are geared 
toward making classroom instruction more rigorous and engaging for students.^ Much research 
has posited that an intensive focus on improving the services and instruction that students 
receive are key pillars of successful school reform.* 

Achieving the Dream could benefit from these efforts by helping colleges develop a 
more systematic approach to identifying promising reforms for improving students’ achieve- 
ment. As in K-12 models, in which principals serve as key instructional leaders. Achieving the 
Dream could encourage senior administrators to gain a deeper knowledge of their colleges’ own 
classroom practices and build momentum for learning about new and improved ways of 
promoting students’ achievement. Additionally, given the wide network that Achieving the 
Dream now encompasses, the initiative could encourage broader sharing of successful practices 
across institutions through professional workgroups or quarterly meetings, with a systematic 
focus on interventions that improve student learning. Such efforts might result in insights about 
what practices can actually change students’ achievement and help colleges develop more 
refined methods for choosing their approaches to improve students’ success. 

• Provide more support to colleges that are experiencing leadership turnover. 

Nearly half of the Round 1 colleges participating in Achieving the Dream experienced 
turnover in their senior leadership, including both presidents and other senior leaders, during 
their five years in the initiative. Such turnover created major difficulties for a number of 
colleges in terms of their progress within the initiative, as they were unable to develop clarity or 
momentum in their plans for improving student success. In fact, those colleges that had the 
greatest difficulties with Achieving the Dream tended to have high levels of turnover among 
their senior administrative staff. 

Given that leadership turnover is a common part of the community college experience. 
Achieving the Dream should look for ways to help support colleges when they confront this 
experience.® Diversifying colleges’ leadership of Achieving the Dream, as suggested above. 



*’Wohls tetter, Datnow, and Park (2008); Wagner et al. (2006); Elmore (2000); Quint (2006). 

^Quint, Bloom, Black, and Stephens (2005). 

*Quint (2006); Bryk et al. (2009); Elmore (2000). 

®Brock et al. (2007); Klein and Takeda-Tinker (2009); American Association of Community Colleges 
(2006). 
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may help to smooth such transitions, as lower-level administrators, faculty, and staff are able to 
carry forward the colleges’ efforts. Similarly, rather than involving only one or two central 
players in the colleges’ work to implement changes, engaging a cadre of senior administrators 
and mid-level managers, such as deans and department heads, might help ensure that colleges 
are able to maintain their momentum if some of their senior leaders leave the institution. 
Finally, the initiative might look for additional ways to support colleges during these transition 
periods, such as providing additional coaching or mentoring, or developing more systematized 
guidelines for how colleges might continue their work during unsteady times. 

• Develop clearer guidelines for colleges’ efforts to address racial, ethnic, and 
income achievement gaps. 

Achieving the Dream proposed that reducing racial, ethnic, and income achievement 
gaps was to be a critical part of colleges’ work in the initiative. This message was a clear focus 
of several of the initiative’s Strategy Institutes, and colleges were asked to report on their efforts 
each year as part of their annual reports. However, while many Round 1 college leaders ex- 
pressed an interest in improving student success, relatively few colleges articulated a clear plan 
for closing achievement gaps across different racial, ethnic, and income groups. Colleges gave a 
number of reasons for this lack of focus, ranging from a desire to focus on other types of 
achievement issues, such as developmental education students’ low success rates, to discomfort 
talking about race or income issues, to a lack of clarity about how to focus on these groups, 
particularly when these students made up a majority of a college’s student population. 

Such findings reveal that Achieving the Dream may need to further clarify its expecta- 
tions for colleges’ efforts to improve equity across racial, ethnic, and income lines and provide 
better guidance for colleges that are undertaking this work. Given that a number of colleges 
served highly disadvantaged populations, the initiative might consider how to structure its 
message with these schools and provide models for developing a more nuanced equity agenda 
in colleges whose students are mostly nonwhite or low-income. 

Additionally, while the initiative encourages colleges to focus on equity, it does not re- 
quire schools to develop this agenda when they join the initiative. If Achieving the Dream is 
serious about having colleges undertake this work, it might consider developing a more syste- 
matic approach for colleges to focus on these gaps throughout their participation in the initia- 
tive. Achieving the Dream might begin requiring colleges to submit a detailed statement of their 
plans to confront achievement gaps during the Kickoff Institute or during their planning year. 
Additionally, the initiative could require colleges to examine and report on their annual student 
outcome trends broken out by race, ethnicity, and income status. Finally, given that a number of 
colleges continued to be nervous about approaching sensitive issues of race and income, the 
initiative might consider providing specific coaching or mentoring to help build stronger 
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leadership around this issue. Indeed, research suggests that institutions without good supports to 
tackle these issues are more likely to avoid these discussions.'” 



Improvements to the Data Analysis Model in Achieving the 
Dream 

• Develop stronger supports for colleges with weak institutional research 
capacity. 

While Achieving the Dream clearly had an effect on most colleges’ data capacity, a 
group of Round 1 colleges continued to have difficulty collecting and analyzing student 
outcomes data throughout their five-year participation in the initiative. These schools were 
hampered by issues such as ineffective data systems, a lack of institutional research (IR) 
personnel, or insufficient capabilities to track and monitor students’ achievement. 

Given these colleges’ sustained challenges and the centrality of data analysis to Achiev- 
ing the Dream’s framework, the initiative should consider more proactive ways of supporting 
colleges that enter with very weak data capacity. Several of these schools noted that their 
smaller sizes, mral locations, or challenging financial situations hindered their ability to im- 
prove their IR capacity. Achieving the Dream might, therefore, consider providing additional 
supports to colleges that have those risk factors. For instance. Achieving the Dream might wish 
to assign its strongest data facilitators to colleges with weak data capacity and provide more 
stmctured training for both the colleges and the coaches who are assisting them. Achieving the 
Dream might also consider offering more advice to financially strapped schools on how to 
support the often high costs associated with hiring new staff or obtaining new data systems, 
particularly the new hardware and software systems that are often used to analyze longitudinal 
data. Finally, Achieving the Dream might consider providing specialized help to those colleges 
that are unable to hire sufficient staff to manage their data needs. This may be particularly 
helpful for colleges that are located in mral areas, as they most often noted challenges with 
hiring skilled IR staff. 

• Incorporate more nuanced measures of student outcomes that relate directly 
to classroom practices and students’ learning and progress. 

While the faculty and staff at the Round 1 colleges analyzed some data on students’ 
achievement, results from the faculty and administrator survey revealed that data were not being 
used as regularly as the initiative leadership had hoped. Additionally, certain types of data, such 



"’Wooldridge and Gooden (2009); Gooden (2010a); Gooden (2010b). 
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as measures that monitored the effect of classroom teaching on students’ learning of specific 
course content, tended to be used more often than the institutional measures, such as graduation 
or persistence rates, that are the focus of Achieving the Dream. Finally, a number of faculty and 
staff noted greater involvement with their colleges’ learning outcomes processes, which are 
designed to establish standards and assessments for what students should learn in particular 
courses tied to program learning outcomes. In fact, many Round 1 colleges were undertaking 
student learning outcomes initiatives in concert with their Achieving the Dream work, often in 
response to their accreditation requirements. However, the fruits of this work were rarely 
captured in Achieving the Dream’s overarching measures of student achievement. 

Given faculty and staffs interest in and involvement with student learning outcomes 
and classroom-level data. Achieving the Dream should consider incorporating such measures 
into its model for institutional improvement. In addition to being more relevant for faculty and 
staff, monitoring students’ progress and completion of particular course sequences could aid 
colleges and the initiative in a number of other ways. For instance, these measures could 
provide a useful means for evaluating colleges’ strategies and interventions, as they are able to 
capture changes in student learning, in addition to persistence and progression. Additionally, 
processes for setting, assessing, and using student learning outcomes as a means of monitoring 
institutional improvement are now key requirements emphasized by all the regional accredita- 
tion agencies, providing a strong incentive for colleges to focus on them. Finally, and perhaps 
most important, student learning outcomes measures could provide interim checks on colleges’ 
progress in improving students’ achievement, as they might capture changes in student out- 
comes that may not be reflected in measures of institutional effectiveness. 



Improvements to the Broad Engagement Model in Achieving the 
Dream 

• Concentrate efforts to involve adjunct faculty and staff. 

One of the most salient weaknesses in the Round 1 colleges’ institutional reforms was 
the integration of adjunct or part-time faculty and staff into the change process. Very few Round 
1 colleges found meaningful ways to bring together and support the development of those 
faculty and staff or to involve them in the strategies they were implementing. This is a critical 
oversight, particularly in light of the fact that adjunct faculty teach a large majority of students 
in developmental or gatekeeper introductory courses. If these part-time faculty members are not 
involved in the colleges’ institutional improvement processes, then it is likely that large swaths 
of the colleges’ student bodies will not be touched by their reforms. 
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While Achieving the Dream currently recommends the involvement of part-time facul- 
ty and staff in the institutional reform process, the initiative should seek out new ways to further 
encourage colleges to pursue this work. This may be a challenge, given that many colleges 
noted that neither their budgets nor adjuncts’ busy schedules allowed for more concentrated 
engagement efforts. However, colleges can learn from the few Round 1 colleges, such as 
Valencia, Patrick Henry, and Paul D. Camp, that found more systematic ways to support the 
inclusion of adjunct and part-time faculty. Some key aspects of their work included finding 
ways to give incentives to adjunct faculty to participate in strategy development, such as 
providing additional stipends or slight pay increases; developing centers devoted to the profes- 
sional development of these faculty; finding ways to integrate part-time and full-time staff; and 
giving key leadership roles to adjunct faculty and staff when implementing the colleges’ 
reforms. 



• Provide in-depth and meaningful professional development to all college 
personnel. 

One of the most critical discoveries in school reform research today is how poorly new 
models of instruction are integrated into classroom practice. Traditionally, most school systems 
have relied on one- or two-day conferences or trainings to teach faculty and staff about new 
teaching methods or reforms." However, studies have revealed that such professional develop- 
ment does little to change these individuals’ everyday practice, as they become subsumed in 
their normal routines with little support for integrating their new learning into their practice.'^ 
Instead, school reformers are now urging that institutions develop systematic professional 
development systems, which include training that is embedded within faculty’s and staff’s 
everyday practice and which utilize supports such as peer mentoring and coaching to create a 
continuous model of improvement." While professional development reforms alone do not 
necessarily lead to improved student achievement," a more systematic approach to training 
teachers has been a key tool that K-12 reformers have used to modify instmctor practices and to 
gauge the effect that such practices have on students’ learning. 

Like most K-12 schools, few of the Round 1 Achieving the Dream colleges engaged in 
the type of systematic professional development that is recommended for truly changing 
practitioners’ work; these colleges tended to focus instead on shorter workshops, conferences. 



"Darling-Hammond et al. (2009); Elmore (2000). 

"Troen and Boles (2003); Darling-Hammond et al. (2009). 

"Stigler and Hiebert (1999); Darling-Hammond et al. (2009); Elmore (2000). 

"Some studies, which have focused on the effects of increased professional development on student out- 
comes, have had disappointing results (Corrin et al., 2008; Garet et al., 2010). However, systematic profession- 
al development and training remains key in other K-12 efforts that have seen some success in increasing 
students’ academic and workforce outcomes (Darling-Hammond et al., 2009). 
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or presentations by various speakers as a way to influence practitioners’ work. While those 
activities may be useful tools for exposing faculty and staff to new educational research, they 
are less likely to influence their classroom or support practices. Achieving the Dream may wish 
to place more emphasis on developing or supporting concrete and systemic professional 
development in community colleges. The initiative could learn from a few of the Round 1 
colleges that implemented these more systematic professional development systems, such as 
Patrick Henry, with its training in collaborative learning. 



Recommendations for Policy Change 

While changes at the institutional level are important, some of the challenges that com- 
munity colleges face are broader, more systemic issues related to funding and policy. Policy- 
makers may wish to consider the following recommendations to support larger positive changes 
in community college students’ success. 

• Continue to strengthen Achieving the Dream’s state policy efforts and connect 
these actions to institution-level reforms. 

Achieving the Dream advocated a broad set of reforms aimed at improving community 
college students’ success, both at the institutional and the state policy levels. While this report 
has focused primarily on institution-level changes, the initiative’s state policy efforts, under the 
direction of Jobs for the Future (JFF), also met with a number of successes. For instance, the 
community college system in the state of Virginia has undertaken a broad revamping of its 
developmental education system, in large part a result of the policy work undertaken in Achiev- 
ing the Dream. Texas is undertaking similar work at the state level in an effort to improve the 
achievement of its developmental education students. 

Continuing to strengthen such state-led reforms has a number of advantages for Achiev- 
ing the Dream’s efforts to improve students’ success. First, as JFF’s work has shown, changes at 
the state policy level can bring about changes at all colleges in a state, thereby ensuring that new 
reforms are implemented by a wide group of institutions rather than a select few that may be 
involved in Achieving the Dream. Second, states represent a major funding source for commu- 
nity colleges, so a focus on improving these institutions at the state level may help facilitate the 
dissemination of critical funding needed to undertake new reforms. States’ economic hold on 
community colleges might also create greater accountability systems and ensure that colleges 
are fully enacting newly recommended changes. Finally, states can utilize existing organiza- 



'^Mills (2010); The Virginia Community Colleges Developmental Education Taskforce (2009). 
''’The Texas Higher Education Coordinating Board (2009). 
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tions, such as statewide councils for presidents, faculty, and staff, as vehicles for engaging 
practitioners in new reforms. Tapping into such systems might also help counteract some of the 
challenges that the initiative has had in bringing these stakeholders into the work. 

Given the important role that state policy reform can have in institutional change. 
Achieving the Dream should continue to focus its efforts in these areas. One avenue for 
strengthening this work further is to develop more open communication channels between 
institutions that are participating in Achieving the Dream and those that are involved in 
changes at the state policy level. While both parties are actively pursuing initiatives to 
improve student achievement, they less often have the opportunity to reflect with one another 
about how such changes are being implemented. Achieving the Dream has a number of 
avenues for promoting such conversations, including providing facilitated time at the Strategy 
Institute or coordinated reflections in newsletters or reports. 

• Provide targeted resources for community college improvement. 

One the greatest challenges facing community colleges today is severely restricted 
finances. Most community colleges survive on thin institutional budgets, especially when 
compared with their richer four-year counterparts. Indeed, community colleges’ strapped 
financial situation is often the reason that they employ large numbers of adjunct faculty, 
who can be easily hired and fired at will. Community colleges’ difficulties have only 
increased since the onset of the U.S. recession in 2007-2008, which has tightened state 
budgets and forced many states to make double-digit reductions in their higher education 
and community college funding. 

In order to accomplish the ambitious agenda that policymakers and foundations are en- 
couraging, more funding is needed to support community colleges’ endeavors. As can be seen 
in this report, many Round 1 colleges struggled to employ full-time faculty and staff and 
support their professional development. Given these challenges and the fact that broad institu- 
tional change also requires significant investments in staff and infrastmcture, policymakers 
should continue to pursue ways to strengthen community colleges’ financial profiles so that 
they can afford these endeavors. 

• Develop a more systematic approach to evaluating community colleges’ 
strategies and interventions. 

Achieving the Dream and its funders have invested numerous resources in helping col- 
leges pilot and scale new interventions aimed at improving student success. Additionally, 



'^Strauss (2009); Blumenstyk, Sander, Schmidt, and Wasley (2008); Davis (2009); Dooley (2009); Graver 
(2009); Clark (2009). 
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through its Strategy Institute and other venues, the initiative has helped colleges network about 
their efforts, with many schools developing very similar approaches to increasing students’ 
achievement. However, despite the commonality in colleges’ interventions, few rigorous 
analyses have been undertaken comparing the effects of these various interventions on students’ 
achievement. 

Given the multitude of colleges now participating in the initiative. Achieving the Dream 
has a strong platform for helping advance the field’s knowledge about how specific reforms 
might affect students’ achievement. With numerous colleges implementing similar strategies, 
the initiative, and the funders and policymakers who support it, might look to systematically test 
how variations in these programs might produce different impacts on students’ achievement. 
For instance, analyzing the differences among several colleges’ supplemental instmction 
programs and their effects on student achievement would provide more information about 
which type of program might work best for improving students’ outcomes. A coordinated effort 
to analyze the differences in colleges’ approaches — and the results of their efforts — would 
provide a substantial boon to both the initiative and community colleges at large given the 
relative dearth of rigorous research that is available on these programs. 



Conclusion 

Achieving the Dream has taken bold steps to improve community college students’ 
achievement, and it has reaped many benefits from the hard work of its partners and participat- 
ing colleges since it was launched in 2004. Many Round 1 colleges have exited their implemen- 
tation years with stronger data capacity and a more nuanced understanding of the challenges 
that their students face. They have also developed a number of strategies to improve students’ 
success and made efforts to move their institutions toward a greater focus on these students’ 
needs. While larger changes in students’ achievement may not yet have been realized. Achiev- 
ing the Dream has begun an unprecedented movement toward helping colleges improve their 
students’ outcomes and develop systems to sustain those efforts over the long term. 

Based on the Round 1 colleges’ experiences, a number of suggestions have been pro- 
vided in this report to further strengthen Achieving the Dream’s model. Bringing faculty and 
staff voices more concretely into colleges’ reform work and focusing more directly on im- 
provements to classroom instmction and services may have important benefits for the next stage 
of the initiative’s work. Given the successes they have already had, the initiative and its partici- 
pating colleges stand poised to move forward with this agenda and to make the changes that are 
needed to help community college students across the country accomplish their goals. 
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Appendix A 

Methodology for Survey of Faculty and Administrators at 
Achieving the Dream Colleges 




To examine patterns of and reasons for data use by faculty and administrators at 
Achieving the Dream colleges, the research team, comprising staff from MDRC and the 
Community College Research Center (CCRC), designed a survey to be delivered using self- 
administered, online instmments.' 

The research team developed two survey instmments through a process involving ex- 
tensive internal review and revision. The instmments were pilot-tested in spring 2007 with 
faculty and administrators at three second-round Achieving the Dream colleges,^ and were 
further revised based on the responses and feedback from institutional researchers at the pilot 
colleges. 

The administrator instmment contains 100 items, and the faculty instmment contains 
101 items, most of which overlap. Both instmments ask questions on three main topics: (1) data 
accessibility, (2) data use and usefulness, and (3) familiarity with and involvement in Achieving 
the Dream. Most items use a seven-point Likert scale, ^ although some are questions requiring 
only a “yes” or “no” response. Both instmments also ask for demographic information about the 
respondent and include an open-ended question about both the use of data at the respondent’s 
college and views on Achieving the Dream. 

The sampling frame included administrators and full-time faculty members at the col- 
leges who joined the Achieving the Dream initiative in Round 1 (in summer 2004) and in 
Round 3 (in summer 2006). Three of the Round 1 colleges declined to participate in the survey, 
leaving 23 Round 1 colleges in five states (Florida, New Mexico, North Carolina, Texas, and 
Virginia) and 13 Round 3 colleges in two states (Pennsylvania and Washington). 

The Human Resources Research Organization (HumRRO) administered the survey. 
HumRRO also participated in the final stages of questionnaire development. HumRRO staff 
were responsible for the Web programming to permit online administration of the survey, 
maintenance of the online survey, and management of the paper versions of the survey and of 
the telephone administration of the survey, and served as the primary contact for survey ques- 
tions and technical difficulties. Finally, HumRRO provided CCRC with an interim data set in 
January 2008, and the final data set in February 2008. 



*The survey methodology and respondent demographics and characteristics described here are from Jen- 
kins and Kerrigan (2008), pp. 9-11. 

^Colleges participating in the demonstration phase of the initiative joined the Achieving the Dream initia- 
tive in four “rounds” — Round 1, summer 2004; Round 2, summer 2005; Round 3, summer 2006; and Round 
4, summer 2007. 

Likert scale is a psychometric scale, often used in questionnaires and surveys, which allows respon- 
dents to express their level of agreement with a particular statement. 
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The research team asked colleges to provide names, titles, and e-mail addresses for full- 
time faculty members and administrators. For the purposes of this survey, “administrator” was 
defined as anyone at the assistant/associate director level and above, but the colleges were 
allowed to decide whom to include in this category. In cases where a college provided the team 
with all nonfaculty positions, the list of administrators was edited based on the job titles that 
were provided. This process generated a list of 2,209 full-time administrators. 

The faculty survey was confined to full-time faculty members, since many colleges do 
not collect contact information on adjunct faculty at the college level. Because some of the 
larger colleges had a large number of faculty, a single-stage sampling procedure was used to 
reduce the number of faculty who were invited to participate in the study. At the colleges with 
faculties larger than 150, the team randomly selected 150 faculty members by automatically 
generating random numbers for each individual and then selecting the first 150, the minimum 
number that the team estimated was needed to ensure statistical rigor. Individuals who were no 
longer affiliated with a given college or were on leave were deleted, and the next person on the 
list was selected as a replacement. This process of replacement continued only during the pre- 
survey phase, when e-mail addresses were being tested; once the survey was deployed, no 
replacements occurred. Through this process, 4,130 full-time faculty members were invited to 
participate in the study. 

The survey was administered beginning in September 2007. To encourage a high re- 
sponse rate, a multiphase administration process was used. It began with a preliminary e-mail 
message designed to give advance notice of the survey and to test the list of e-mail addresses 
that the research team had received from the colleges. The next communication, one week later, 
was an e-mail message with the URL for the survey and a unique password for each respondent. 
The third e-mail message was sent as a follow-up one week later to everyone who had not yet 
responded. A fourth e-mail message was sent one month after the initial invitation. One week 
later, letters with a hard copy of the survey enclosed were mailed to all nonrespondents. The 
mailing of the hard copy letters coincided with a request to the college presidents to encourage 
their faculty and administrators to complete the survey. Phone administration of the survey to 
those who had not yet responded began just over two months after the initial e-mail message 
was sent. At that time, the colleges were approaching winter break, so efforts to contact the 
faculty and administrators at the colleges were suspended until after the new year. A final e- 
mail message went to all nonrespondents, and the survey was concluded with a final one-week 
phone administration in the week following the February 2008 Achieving the Dream Strategy 
Institute, which was attended by teams from every participating college. 

The survey achieved a response rate of 60 percent from the faculty surveyed and a 73 
percent response rate from the administrators. Response rates varied considerably by college. 
Responses to the faculty survey ranged from a low of 37 percent at one college to a high of 85 
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percent at another. College response rates to the administrator survey ranged from 57 percent to 
93 percent. 

Respondent demographics of faculty and administrators were fairly similar across col- 
leges. Of the faculty respondents, 56 percent were female, and of the administrators, 62 percent 
were female. The majority of faculty and administrator survey respondents, more than 78 
percent, identified themselves as white. The next largest group comprised those who identified 
themselves as Hispanic/Latino/Spanish; they made up 12 percent of the faculty and 15 percent 
of the administrators. Black, Asian or Pacific Islander, and American Indian or Alaskan Native 
faculty accounted for 8 percent, 3 percent, and 2 percent of the faculty respondents, respective- 
ly. Black administrators made up 12 percent of the administrator respondents. Asian or Pacific 
Islanders and American Indian or Alaskan Natives made up 3 percent and 1 percent of adminis- 
trator respondents, respectively. Ten percent of faculty and 8 percent of administrators identified 
themselves as “other.” 

More than 62 percent of the faculty who completed the survey taught in academic trans- 
fer/general education areas. Administrators involved in academic affairs and instruction made 
up 20 percent of the respondents, and those in student affairs/student services accounted for 1 8 
percent of the respondents. The institutional research staff accounted for just over 3 percent of 
the respondents. 
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Appendix B 

Technical Appendix 




Data 



JBL Associates, a higher education consulting firm, is compiling the Achieving the 
Dream database with student-level data from all of the participating colleges. These data, used 
to measure progress on Achieving the Dream’s five student outcome indicators,' include 
demographic characteristics, transcript information, and associate’s degrees and certificates for 
fall cohorts of credential-seeking students enrolling in college for the first time at each college. 
The database includes students who are seeking short-term certificates, longer-term certificates, 
and associate’s degrees both full time and part time. This is the population tracked by the 
Integrated Postsecondary Education Data System (IPEDS), with the addition of part-time 
students. The colleges have submitted regular updates on each cohort, which make it possible to 
track the outcomes for the students over the course of the initiative. 



Regression Model Used to Estimate Mean Student Outcomes 

Student-level data from the Achieving the Dream database are aggregated to produce 
institutional averages. Ordinary least squares regression is employed to estimate the following 
model: 



y. =a + P2QQ5.+ P2QQ6.+ piQQl .+yT +X. A + V O + s. 
it 1 z 2 z 3 z it s ii 



It 



where 



yu = the average of the two-year student outcome measure for institution i 

2005i = 1 if the institutional average i is part of the post-initiative cohort of 2005, 
and 0 otherwise 

r= a time trend employed to account for trends in institutional outcomes es- 
tablished prior to the start of the initiative (so that changes in the post- 
initiative period are not overshadowed by pre-initiative trends) 



'The five measures of student success in Achieving the Dream are (1) completion of developmental 
courses and progression to credit-bearing courses; (2) completion of so-called gatekeeper courses, 
including introductory college courses in English and math; (3) completion of attempted courses with a 
grade of “C” or better; (4) persistence from semester to semester and from year to year; and (5) attain- 
ment of college credentials. 
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Xit = a matrix of average institutional student characteristics, including the 
proportion of males, age at cohort entry, the proportion of students in 
several race/ethnicity categories, and the proportion of students who 
have nonresident alien status 

Vi = a matrix of state indicator variables 

Sit = a random error term for observation i, which is assumed to vary by col- 
lege — that is. Sit = Vi + Ut, where Ut is assumed to be independently 
and identically distributed 



Alternative Trend Breaks 

Given that Achieving the Dream had a long “ramp-up” phase, it is important to deter- 
mine when to mark the start of the Achieving the Dream period. While the analyses in Chapter 
8 use 2005 to represent the official start of Achieving the Dream, earlier chapters in this report 
show that the actual implementation of Achieving the Dream varied widely. As a result, the 
period after 2005 may be too early for many colleges and possibly too late for others (if they 
had started to implement changes prior to being selected to join the Achieving the Dream 
initiative). This issue is addressed in two ways: 

1. The provision of the Achieving the Dream grants in 2004 is used as a proxy 
for the start of the Achieving the Dream initiative in 2005, as done in the 
Achieving the Dream baseline report.^ 

2. Since the precise timing of the start of the initiative is unknown, data on stu- 
dent outcomes were used to identify when a stmctural break in trend occurs 
as opposed to imposing a period for the break.^ Since implementation of 
Achieving the Dream varied by college, as indicated by the field research 
findings in this report, student-level data are examined to indicate when im- 
plementation of the Achieving the Dream model may have occurred. 

The results from the stmctural break analysis suggest that the initiative was imple- 
mented in 2005 (see Appendix Table B.l). The analyses, therefore, use 2002-2004 as the pre- 
initiative period. 



^Brock et al. (2007). 

^Piehl, Cooper, Braga, and Kennedy (1999); Andrews (1993). 
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Achieving the Dream: Community Colleges Count 
Appendix Table B.l 

Alternative Trend Breaks in Interim Performance Indicators 



Performance Indicators 


Maximum 
Test Statistic 


Year of 
Maximum 


Referred to any developmental courses® (%) 


Developmental math 


1.6 


2005 


Attempted course within first year 


1.3 


2005 


Completed requirement within 1 year 


2.7 


2006 


Developmental English 


1.3 


2004 


Attempted course within first year 


4.1 


2005 


Completed requirement within 1 year 


7.3 


2005 


Developmental reading 


2.3 


2005 


Attempted course within first year 


0.5 


2006 


Completed requirement within 1 year 


1.9 


2005 


Successfullv completed 2atekeeper courses®’’’ (%) 

Gatekeeper math course within first year, among students referred to 
developmental math' 


6.2 


2005 


Gatekeeper English course within first year, among students referred to 
developmental English' 


10.4 


2005 


Course completion'’ 


Attempted credits within first year 


0.0 


2005 


Completed credits within first year 


7.7 


2004 


Ratio of completed credits to attempted credits within first year (%) 


5.7 


2004 


Completed 80 percent or more of attempted credits within first year (%) 


6.1 


2004 


Persistence (%) 


Enrolled in first semester after initial term of enrollment 


3.8 


2006 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for this table used all available data for sample members in the fall 2002 through fall 
2007 cohorts at Achieving the Dream Round 1 colleges to derive averages of the performance indicators . The 
sample size used in the table is N = 421,202, years 2002 through 2007. Figures control for student 
characteristics and state. The chi-squared critical values for testing for a break in trend with appropriate 
trimming are 5.95 at the 10 percent level of statistical significance, 7.4 at the 5 percent level, and 10.8 at the 1 
percent level. 

^Grade of "C" or better must be earned to successfully complete a course. 

'’Gatekeeper math is the first college-level math course at the college; gatekeeper English is the first 
college-level English course at the college. 

‘’Calculations used data for sample members who have developmental referral data. 

‘'Completion with a grade of "D" or better is reported for this block of outcomes. Credits completed with a 
grade of "C" or better was not introduced as a variable until the 2007 JBL submission guide was released. 
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